2023-04-21 01:53:33,408 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de 2023-04-21 01:53:33,426 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-21 01:53:33,445 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-21 01:53:33,445 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/cluster_95a6cece-d06d-0ad6-c9d7-e0ee9e760277, deleteOnExit=true 2023-04-21 01:53:33,445 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-21 01:53:33,446 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/test.cache.data in system properties and HBase conf 2023-04-21 01:53:33,446 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/hadoop.tmp.dir in system properties and HBase conf 2023-04-21 01:53:33,447 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/hadoop.log.dir in system properties and HBase conf 2023-04-21 01:53:33,447 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-21 01:53:33,448 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-21 01:53:33,448 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-21 01:53:33,569 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-21 01:53:33,999 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-21 01:53:34,005 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-21 01:53:34,005 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-21 01:53:34,006 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-21 01:53:34,006 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-21 01:53:34,007 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-21 01:53:34,007 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-21 01:53:34,008 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-21 01:53:34,008 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-21 01:53:34,009 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-21 01:53:34,009 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/nfs.dump.dir in system properties and HBase conf 2023-04-21 01:53:34,010 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/java.io.tmpdir in system properties and HBase conf 2023-04-21 01:53:34,010 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-21 01:53:34,010 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-21 01:53:34,011 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-21 01:53:34,632 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-21 01:53:34,636 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-21 01:53:34,928 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-21 01:53:35,116 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-21 01:53:35,137 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-21 01:53:35,180 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-21 01:53:35,211 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/java.io.tmpdir/Jetty_localhost_40465_hdfs____.u6ox27/webapp 2023-04-21 01:53:35,377 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40465 2023-04-21 01:53:35,416 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-21 01:53:35,416 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-21 01:53:35,913 WARN [Listener at localhost/36883] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-21 01:53:36,049 WARN [Listener at localhost/36883] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-21 01:53:36,072 WARN [Listener at localhost/36883] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-21 01:53:36,080 INFO [Listener at localhost/36883] log.Slf4jLog(67): jetty-6.1.26 2023-04-21 01:53:36,085 INFO [Listener at localhost/36883] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/java.io.tmpdir/Jetty_localhost_39071_datanode____.6jri9h/webapp 2023-04-21 01:53:36,195 INFO [Listener at localhost/36883] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39071 2023-04-21 01:53:36,591 WARN [Listener at localhost/39939] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-21 01:53:37,194 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x844798935bc22312: Processing first storage report for DS-173379a7-7b65-441c-854e-6f977505d52b from datanode 42d80742-5c6e-423e-b2cf-6734fc36de52 2023-04-21 01:53:37,194 DEBUG [Listener at localhost/39939] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de 2023-04-21 01:53:37,197 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x844798935bc22312: from storage DS-173379a7-7b65-441c-854e-6f977505d52b node DatanodeRegistration(127.0.0.1:40433, datanodeUuid=42d80742-5c6e-423e-b2cf-6734fc36de52, infoPort=46729, infoSecurePort=0, ipcPort=39939, storageInfo=lv=-57;cid=testClusterID;nsid=426445860;c=1682042014715), blocks: 0, hasStaleStorage: true, processing time: 3 msecs, invalidatedBlocks: 0 2023-04-21 01:53:37,197 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x844798935bc22312: Processing first storage report for DS-e13eb5df-f75f-4915-ac8d-e650fc0dab11 from datanode 42d80742-5c6e-423e-b2cf-6734fc36de52 2023-04-21 01:53:37,197 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x844798935bc22312: from storage DS-e13eb5df-f75f-4915-ac8d-e650fc0dab11 node DatanodeRegistration(127.0.0.1:40433, datanodeUuid=42d80742-5c6e-423e-b2cf-6734fc36de52, infoPort=46729, infoSecurePort=0, ipcPort=39939, storageInfo=lv=-57;cid=testClusterID;nsid=426445860;c=1682042014715), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-21 01:53:37,276 INFO [Listener at localhost/39939] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/cluster_95a6cece-d06d-0ad6-c9d7-e0ee9e760277/zookeeper_0, clientPort=58383, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/cluster_95a6cece-d06d-0ad6-c9d7-e0ee9e760277/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/cluster_95a6cece-d06d-0ad6-c9d7-e0ee9e760277/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-21 01:53:37,289 INFO [Listener at localhost/39939] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=58383 2023-04-21 01:53:37,296 INFO [Listener at localhost/39939] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-21 01:53:37,299 INFO [Listener at localhost/39939] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-21 01:53:37,998 INFO [Listener at localhost/39939] util.FSUtils(471): Created version file at hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58 with version=8 2023-04-21 01:53:37,998 INFO [Listener at localhost/39939] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/hbase-staging 2023-04-21 01:53:38,305 INFO [Listener at localhost/39939] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-21 01:53:38,819 INFO [Listener at localhost/39939] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-21 01:53:38,852 INFO [Listener at localhost/39939] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-21 01:53:38,852 INFO [Listener at localhost/39939] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-21 01:53:38,853 INFO [Listener at localhost/39939] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-21 01:53:38,853 INFO [Listener at localhost/39939] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-21 01:53:38,853 INFO [Listener at localhost/39939] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-21 01:53:39,013 INFO [Listener at localhost/39939] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-21 01:53:39,086 DEBUG [Listener at localhost/39939] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-21 01:53:39,183 INFO [Listener at localhost/39939] ipc.NettyRpcServer(120): Bind to /172.31.14.131:35449 2023-04-21 01:53:39,194 INFO [Listener at localhost/39939] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-21 01:53:39,196 INFO [Listener at localhost/39939] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-21 01:53:39,218 INFO [Listener at localhost/39939] zookeeper.RecoverableZooKeeper(93): Process identifier=master:35449 connecting to ZooKeeper ensemble=127.0.0.1:58383 2023-04-21 01:53:39,261 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:354490x0, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-21 01:53:39,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:35449-0x101a46c3fae0000 connected 2023-04-21 01:53:39,293 DEBUG [Listener at localhost/39939] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-21 01:53:39,294 DEBUG [Listener at localhost/39939] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-21 01:53:39,298 DEBUG [Listener at localhost/39939] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-21 01:53:39,310 DEBUG [Listener at localhost/39939] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35449 2023-04-21 01:53:39,311 DEBUG [Listener at localhost/39939] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35449 2023-04-21 01:53:39,311 DEBUG [Listener at localhost/39939] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35449 2023-04-21 01:53:39,312 DEBUG [Listener at localhost/39939] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35449 2023-04-21 01:53:39,315 DEBUG [Listener at localhost/39939] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35449 2023-04-21 01:53:39,324 INFO [Listener at localhost/39939] master.HMaster(444): hbase.rootdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58, hbase.cluster.distributed=false 2023-04-21 01:53:39,400 INFO [Listener at localhost/39939] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-21 01:53:39,400 INFO [Listener at localhost/39939] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-21 01:53:39,400 INFO [Listener at localhost/39939] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-21 01:53:39,400 INFO [Listener at localhost/39939] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-21 01:53:39,401 INFO [Listener at localhost/39939] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-21 01:53:39,401 INFO [Listener at localhost/39939] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-21 01:53:39,406 INFO [Listener at localhost/39939] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-21 01:53:39,409 INFO [Listener at localhost/39939] ipc.NettyRpcServer(120): Bind to /172.31.14.131:40131 2023-04-21 01:53:39,411 INFO [Listener at localhost/39939] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-21 01:53:39,417 DEBUG [Listener at localhost/39939] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-21 01:53:39,418 INFO [Listener at localhost/39939] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-21 01:53:39,421 INFO [Listener at localhost/39939] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-21 01:53:39,423 INFO [Listener at localhost/39939] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:40131 connecting to ZooKeeper ensemble=127.0.0.1:58383 2023-04-21 01:53:39,426 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:401310x0, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-21 01:53:39,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:40131-0x101a46c3fae0001 connected 2023-04-21 01:53:39,428 DEBUG [Listener at localhost/39939] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-21 01:53:39,429 DEBUG [Listener at localhost/39939] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-21 01:53:39,430 DEBUG [Listener at localhost/39939] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-21 01:53:39,430 DEBUG [Listener at localhost/39939] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40131 2023-04-21 01:53:39,430 DEBUG [Listener at localhost/39939] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40131 2023-04-21 01:53:39,432 DEBUG [Listener at localhost/39939] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40131 2023-04-21 01:53:39,436 DEBUG [Listener at localhost/39939] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40131 2023-04-21 01:53:39,436 DEBUG [Listener at localhost/39939] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40131 2023-04-21 01:53:39,438 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,35449,1682042018145 2023-04-21 01:53:39,449 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-21 01:53:39,450 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,35449,1682042018145 2023-04-21 01:53:39,472 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-21 01:53:39,472 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-21 01:53:39,472 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-21 01:53:39,473 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-21 01:53:39,475 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,35449,1682042018145 from backup master directory 2023-04-21 01:53:39,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-21 01:53:39,478 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,35449,1682042018145 2023-04-21 01:53:39,478 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-21 01:53:39,479 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-21 01:53:39,479 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,35449,1682042018145 2023-04-21 01:53:39,482 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-21 01:53:39,484 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-21 01:53:39,573 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/hbase.id with ID: 8f8af8c0-6174-45db-be9a-ca59a6391305 2023-04-21 01:53:39,618 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-21 01:53:39,635 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-21 01:53:39,679 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x2881fe27 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:53:39,716 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bfb9b11, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:53:39,742 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-21 01:53:39,745 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-21 01:53:39,767 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-21 01:53:39,767 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-21 01:53:39,770 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:53:39,774 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:53:39,776 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-21 01:53:39,812 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/MasterData/data/master/store-tmp 2023-04-21 01:53:39,842 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-21 01:53:39,842 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-21 01:53:39,842 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-21 01:53:39,842 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-21 01:53:39,842 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-21 01:53:39,842 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-21 01:53:39,842 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-21 01:53:39,843 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-21 01:53:39,845 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/MasterData/WALs/jenkins-hbase4.apache.org,35449,1682042018145 2023-04-21 01:53:39,870 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35449%2C1682042018145, suffix=, logDir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/MasterData/WALs/jenkins-hbase4.apache.org,35449,1682042018145, archiveDir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/MasterData/oldWALs, maxLogs=10 2023-04-21 01:53:39,931 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40433,DS-173379a7-7b65-441c-854e-6f977505d52b,DISK] 2023-04-21 01:53:39,941 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:53:40,011 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/MasterData/WALs/jenkins-hbase4.apache.org,35449,1682042018145/jenkins-hbase4.apache.org%2C35449%2C1682042018145.1682042019881 2023-04-21 01:53:40,012 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40433,DS-173379a7-7b65-441c-854e-6f977505d52b,DISK]] 2023-04-21 01:53:40,013 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-21 01:53:40,014 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-21 01:53:40,018 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-21 01:53:40,020 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-21 01:53:40,083 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-21 01:53:40,091 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-21 01:53:40,124 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-21 01:53:40,139 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:53:40,145 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-21 01:53:40,146 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-21 01:53:40,164 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-21 01:53:40,169 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-21 01:53:40,170 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60386474, jitterRate=-0.10017141699790955}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-21 01:53:40,171 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-21 01:53:40,172 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-21 01:53:40,196 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-21 01:53:40,197 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-21 01:53:40,200 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-21 01:53:40,202 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-21 01:53:40,240 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 37 msec 2023-04-21 01:53:40,240 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-21 01:53:40,269 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-21 01:53:40,276 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-21 01:53:40,303 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-21 01:53:40,307 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-21 01:53:40,310 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-21 01:53:40,316 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-21 01:53:40,320 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-21 01:53:40,323 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-21 01:53:40,325 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-21 01:53:40,326 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-21 01:53:40,342 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-21 01:53:40,349 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-21 01:53:40,349 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-21 01:53:40,349 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-21 01:53:40,351 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,35449,1682042018145, sessionid=0x101a46c3fae0000, setting cluster-up flag (Was=false) 2023-04-21 01:53:40,368 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-21 01:53:40,374 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-21 01:53:40,375 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,35449,1682042018145 2023-04-21 01:53:40,381 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-21 01:53:40,387 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-21 01:53:40,388 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,35449,1682042018145 2023-04-21 01:53:40,396 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.hbase-snapshot/.tmp 2023-04-21 01:53:40,443 INFO [RS:0;jenkins-hbase4:40131] regionserver.HRegionServer(952): ClusterId : 8f8af8c0-6174-45db-be9a-ca59a6391305 2023-04-21 01:53:40,447 DEBUG [RS:0;jenkins-hbase4:40131] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-21 01:53:40,458 DEBUG [RS:0;jenkins-hbase4:40131] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-21 01:53:40,458 DEBUG [RS:0;jenkins-hbase4:40131] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-21 01:53:40,462 DEBUG [RS:0;jenkins-hbase4:40131] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-21 01:53:40,464 DEBUG [RS:0;jenkins-hbase4:40131] zookeeper.ReadOnlyZKClient(139): Connect 0x79c8271d to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:53:40,474 DEBUG [RS:0;jenkins-hbase4:40131] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@770fc654, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:53:40,476 DEBUG [RS:0;jenkins-hbase4:40131] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@73304c6d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-21 01:53:40,508 DEBUG [RS:0;jenkins-hbase4:40131] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:40131 2023-04-21 01:53:40,512 INFO [RS:0;jenkins-hbase4:40131] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-21 01:53:40,512 INFO [RS:0;jenkins-hbase4:40131] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-21 01:53:40,513 DEBUG [RS:0;jenkins-hbase4:40131] regionserver.HRegionServer(1023): About to register with Master. 2023-04-21 01:53:40,515 INFO [RS:0;jenkins-hbase4:40131] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,35449,1682042018145 with isa=jenkins-hbase4.apache.org/172.31.14.131:40131, startcode=1682042019399 2023-04-21 01:53:40,533 DEBUG [RS:0;jenkins-hbase4:40131] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-21 01:53:40,538 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-21 01:53:40,548 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-21 01:53:40,548 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-21 01:53:40,549 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-21 01:53:40,549 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-21 01:53:40,549 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-21 01:53:40,549 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-21 01:53:40,549 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-21 01:53:40,549 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-21 01:53:40,551 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682042050551 2023-04-21 01:53:40,554 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-21 01:53:40,560 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-21 01:53:40,561 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-21 01:53:40,566 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-21 01:53:40,567 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-21 01:53:40,575 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-21 01:53:40,576 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-21 01:53:40,576 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-21 01:53:40,576 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-21 01:53:40,577 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-21 01:53:40,579 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-21 01:53:40,581 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-21 01:53:40,582 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-21 01:53:40,584 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-21 01:53:40,585 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-21 01:53:40,587 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682042020586,5,FailOnTimeoutGroup] 2023-04-21 01:53:40,588 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682042020587,5,FailOnTimeoutGroup] 2023-04-21 01:53:40,588 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-21 01:53:40,588 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-21 01:53:40,589 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-21 01:53:40,589 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-21 01:53:40,593 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47719, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-21 01:53:40,610 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:40,625 DEBUG [RS:0;jenkins-hbase4:40131] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58 2023-04-21 01:53:40,625 DEBUG [RS:0;jenkins-hbase4:40131] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:36883 2023-04-21 01:53:40,625 DEBUG [RS:0;jenkins-hbase4:40131] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-21 01:53:40,632 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-21 01:53:40,633 DEBUG [RS:0;jenkins-hbase4:40131] zookeeper.ZKUtil(162): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:40,633 WARN [RS:0;jenkins-hbase4:40131] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-21 01:53:40,634 INFO [RS:0;jenkins-hbase4:40131] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-21 01:53:40,635 DEBUG [RS:0;jenkins-hbase4:40131] regionserver.HRegionServer(1947): logDir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/WALs/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:40,635 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,40131,1682042019399] 2023-04-21 01:53:40,644 DEBUG [RS:0;jenkins-hbase4:40131] zookeeper.ZKUtil(162): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:40,658 DEBUG [RS:0;jenkins-hbase4:40131] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-21 01:53:40,669 INFO [RS:0;jenkins-hbase4:40131] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-21 01:53:40,692 INFO [RS:0;jenkins-hbase4:40131] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-21 01:53:40,696 INFO [RS:0;jenkins-hbase4:40131] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-21 01:53:40,696 INFO [RS:0;jenkins-hbase4:40131] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-21 01:53:40,698 INFO [RS:0;jenkins-hbase4:40131] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-21 01:53:40,706 INFO [RS:0;jenkins-hbase4:40131] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-21 01:53:40,706 DEBUG [RS:0;jenkins-hbase4:40131] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-21 01:53:40,707 DEBUG [RS:0;jenkins-hbase4:40131] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-21 01:53:40,707 DEBUG [RS:0;jenkins-hbase4:40131] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-21 01:53:40,707 DEBUG [RS:0;jenkins-hbase4:40131] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-21 01:53:40,707 DEBUG [RS:0;jenkins-hbase4:40131] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-21 01:53:40,708 DEBUG [RS:0;jenkins-hbase4:40131] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-21 01:53:40,708 DEBUG [RS:0;jenkins-hbase4:40131] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-21 01:53:40,708 DEBUG [RS:0;jenkins-hbase4:40131] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-21 01:53:40,708 DEBUG [RS:0;jenkins-hbase4:40131] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-21 01:53:40,708 DEBUG [RS:0;jenkins-hbase4:40131] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-21 01:53:40,709 INFO [RS:0;jenkins-hbase4:40131] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-21 01:53:40,710 INFO [RS:0;jenkins-hbase4:40131] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-21 01:53:40,710 INFO [RS:0;jenkins-hbase4:40131] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-21 01:53:40,728 INFO [RS:0;jenkins-hbase4:40131] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-21 01:53:40,730 INFO [RS:0;jenkins-hbase4:40131] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40131,1682042019399-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-21 01:53:40,747 INFO [RS:0;jenkins-hbase4:40131] regionserver.Replication(203): jenkins-hbase4.apache.org,40131,1682042019399 started 2023-04-21 01:53:40,748 INFO [RS:0;jenkins-hbase4:40131] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,40131,1682042019399, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:40131, sessionid=0x101a46c3fae0001 2023-04-21 01:53:40,748 DEBUG [RS:0;jenkins-hbase4:40131] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-21 01:53:40,748 DEBUG [RS:0;jenkins-hbase4:40131] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:40,748 DEBUG [RS:0;jenkins-hbase4:40131] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40131,1682042019399' 2023-04-21 01:53:40,748 DEBUG [RS:0;jenkins-hbase4:40131] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:40,749 DEBUG [RS:0;jenkins-hbase4:40131] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:40,750 DEBUG [RS:0;jenkins-hbase4:40131] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-21 01:53:40,750 DEBUG [RS:0;jenkins-hbase4:40131] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-21 01:53:40,750 DEBUG [RS:0;jenkins-hbase4:40131] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:40,750 DEBUG [RS:0;jenkins-hbase4:40131] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40131,1682042019399' 2023-04-21 01:53:40,750 DEBUG [RS:0;jenkins-hbase4:40131] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-21 01:53:40,750 DEBUG [RS:0;jenkins-hbase4:40131] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-21 01:53:40,751 DEBUG [RS:0;jenkins-hbase4:40131] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-21 01:53:40,751 INFO [RS:0;jenkins-hbase4:40131] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-21 01:53:40,751 INFO [RS:0;jenkins-hbase4:40131] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-21 01:53:40,862 INFO [RS:0;jenkins-hbase4:40131] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40131%2C1682042019399, suffix=, logDir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/WALs/jenkins-hbase4.apache.org,40131,1682042019399, archiveDir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/oldWALs, maxLogs=32 2023-04-21 01:53:40,884 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40433,DS-173379a7-7b65-441c-854e-6f977505d52b,DISK] 2023-04-21 01:53:40,888 INFO [RS:0;jenkins-hbase4:40131] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/WALs/jenkins-hbase4.apache.org,40131,1682042019399/jenkins-hbase4.apache.org%2C40131%2C1682042019399.1682042020864 2023-04-21 01:53:40,889 DEBUG [RS:0;jenkins-hbase4:40131] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40433,DS-173379a7-7b65-441c-854e-6f977505d52b,DISK]] 2023-04-21 01:53:41,008 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-21 01:53:41,009 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-21 01:53:41,010 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58 2023-04-21 01:53:41,033 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-21 01:53:41,040 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-21 01:53:41,045 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740/info 2023-04-21 01:53:41,046 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-21 01:53:41,048 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:53:41,049 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-21 01:53:41,053 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740/rep_barrier 2023-04-21 01:53:41,053 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-21 01:53:41,054 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:53:41,054 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-21 01:53:41,057 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740/table 2023-04-21 01:53:41,058 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-21 01:53:41,059 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:53:41,061 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740 2023-04-21 01:53:41,062 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740 2023-04-21 01:53:41,067 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-21 01:53:41,070 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-21 01:53:41,074 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-21 01:53:41,076 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59206340, jitterRate=-0.11775678396224976}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-21 01:53:41,076 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-21 01:53:41,076 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-21 01:53:41,076 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-21 01:53:41,076 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-21 01:53:41,076 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-21 01:53:41,076 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-21 01:53:41,078 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-21 01:53:41,078 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-21 01:53:41,084 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-21 01:53:41,084 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-21 01:53:41,098 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-21 01:53:41,111 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-21 01:53:41,113 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-21 01:53:41,265 DEBUG [jenkins-hbase4:35449] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-21 01:53:41,269 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,40131,1682042019399, state=OPENING 2023-04-21 01:53:41,276 DEBUG [PEWorker-4] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-21 01:53:41,281 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-21 01:53:41,282 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-21 01:53:41,285 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,40131,1682042019399}] 2023-04-21 01:53:41,470 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:41,472 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-21 01:53:41,476 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33422, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-21 01:53:41,490 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-21 01:53:41,491 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-21 01:53:41,495 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40131%2C1682042019399.meta, suffix=.meta, logDir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/WALs/jenkins-hbase4.apache.org,40131,1682042019399, archiveDir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/oldWALs, maxLogs=32 2023-04-21 01:53:41,515 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40433,DS-173379a7-7b65-441c-854e-6f977505d52b,DISK] 2023-04-21 01:53:41,519 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/WALs/jenkins-hbase4.apache.org,40131,1682042019399/jenkins-hbase4.apache.org%2C40131%2C1682042019399.meta.1682042021496.meta 2023-04-21 01:53:41,520 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40433,DS-173379a7-7b65-441c-854e-6f977505d52b,DISK]] 2023-04-21 01:53:41,520 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-21 01:53:41,522 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-21 01:53:41,540 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-21 01:53:41,545 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-21 01:53:41,551 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-21 01:53:41,552 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-21 01:53:41,552 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-21 01:53:41,552 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-21 01:53:41,555 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-21 01:53:41,557 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740/info 2023-04-21 01:53:41,557 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740/info 2023-04-21 01:53:41,558 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-21 01:53:41,559 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:53:41,559 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-21 01:53:41,561 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740/rep_barrier 2023-04-21 01:53:41,561 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740/rep_barrier 2023-04-21 01:53:41,562 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-21 01:53:41,563 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:53:41,564 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-21 01:53:41,565 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740/table 2023-04-21 01:53:41,565 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740/table 2023-04-21 01:53:41,566 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-21 01:53:41,567 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:53:41,569 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740 2023-04-21 01:53:41,572 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740 2023-04-21 01:53:41,577 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-21 01:53:41,579 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-21 01:53:41,581 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67865865, jitterRate=0.01128019392490387}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-21 01:53:41,581 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-21 01:53:41,591 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1682042021462 2023-04-21 01:53:41,614 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-21 01:53:41,614 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-21 01:53:41,616 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,40131,1682042019399, state=OPEN 2023-04-21 01:53:41,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-21 01:53:41,619 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-21 01:53:41,625 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-21 01:53:41,626 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,40131,1682042019399 in 335 msec 2023-04-21 01:53:41,631 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-21 01:53:41,632 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 530 msec 2023-04-21 01:53:41,637 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.1880 sec 2023-04-21 01:53:41,638 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1682042021637, completionTime=-1 2023-04-21 01:53:41,638 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-21 01:53:41,639 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-21 01:53:41,710 DEBUG [hconnection-0x4dbb6610-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:53:41,714 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33438, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:53:41,733 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-21 01:53:41,733 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682042081733 2023-04-21 01:53:41,733 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682042141733 2023-04-21 01:53:41,733 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 94 msec 2023-04-21 01:53:41,757 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35449,1682042018145-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-21 01:53:41,757 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35449,1682042018145-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-21 01:53:41,757 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35449,1682042018145-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-21 01:53:41,759 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:35449, period=300000, unit=MILLISECONDS is enabled. 2023-04-21 01:53:41,759 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-21 01:53:41,766 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-21 01:53:41,776 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-21 01:53:41,777 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-21 01:53:41,787 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-21 01:53:41,789 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-21 01:53:41,792 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-21 01:53:41,817 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/hbase/namespace/cc2eba61dc5e0ef952a659f625b99f73 2023-04-21 01:53:41,819 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/hbase/namespace/cc2eba61dc5e0ef952a659f625b99f73 empty. 2023-04-21 01:53:41,820 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/hbase/namespace/cc2eba61dc5e0ef952a659f625b99f73 2023-04-21 01:53:41,820 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-21 01:53:42,248 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-21 01:53:42,251 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => cc2eba61dc5e0ef952a659f625b99f73, NAME => 'hbase:namespace,,1682042021776.cc2eba61dc5e0ef952a659f625b99f73.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp 2023-04-21 01:53:42,664 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682042021776.cc2eba61dc5e0ef952a659f625b99f73.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-21 01:53:42,664 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing cc2eba61dc5e0ef952a659f625b99f73, disabling compactions & flushes 2023-04-21 01:53:42,665 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682042021776.cc2eba61dc5e0ef952a659f625b99f73. 2023-04-21 01:53:42,665 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682042021776.cc2eba61dc5e0ef952a659f625b99f73. 2023-04-21 01:53:42,665 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682042021776.cc2eba61dc5e0ef952a659f625b99f73. after waiting 0 ms 2023-04-21 01:53:42,665 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682042021776.cc2eba61dc5e0ef952a659f625b99f73. 2023-04-21 01:53:42,665 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682042021776.cc2eba61dc5e0ef952a659f625b99f73. 2023-04-21 01:53:42,665 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for cc2eba61dc5e0ef952a659f625b99f73: 2023-04-21 01:53:42,671 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-21 01:53:42,690 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1682042021776.cc2eba61dc5e0ef952a659f625b99f73.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682042022675"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682042022675"}]},"ts":"1682042022675"} 2023-04-21 01:53:42,723 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-21 01:53:42,725 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-21 01:53:42,731 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042022726"}]},"ts":"1682042022726"} 2023-04-21 01:53:42,736 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-21 01:53:42,745 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=cc2eba61dc5e0ef952a659f625b99f73, ASSIGN}] 2023-04-21 01:53:42,748 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=cc2eba61dc5e0ef952a659f625b99f73, ASSIGN 2023-04-21 01:53:42,750 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=cc2eba61dc5e0ef952a659f625b99f73, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40131,1682042019399; forceNewPlan=false, retain=false 2023-04-21 01:53:42,901 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=cc2eba61dc5e0ef952a659f625b99f73, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:42,901 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1682042021776.cc2eba61dc5e0ef952a659f625b99f73.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682042022900"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682042022900"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682042022900"}]},"ts":"1682042022900"} 2023-04-21 01:53:42,906 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure cc2eba61dc5e0ef952a659f625b99f73, server=jenkins-hbase4.apache.org,40131,1682042019399}] 2023-04-21 01:53:43,067 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1682042021776.cc2eba61dc5e0ef952a659f625b99f73. 2023-04-21 01:53:43,068 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => cc2eba61dc5e0ef952a659f625b99f73, NAME => 'hbase:namespace,,1682042021776.cc2eba61dc5e0ef952a659f625b99f73.', STARTKEY => '', ENDKEY => ''} 2023-04-21 01:53:43,069 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace cc2eba61dc5e0ef952a659f625b99f73 2023-04-21 01:53:43,069 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682042021776.cc2eba61dc5e0ef952a659f625b99f73.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-21 01:53:43,069 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for cc2eba61dc5e0ef952a659f625b99f73 2023-04-21 01:53:43,069 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for cc2eba61dc5e0ef952a659f625b99f73 2023-04-21 01:53:43,072 INFO [StoreOpener-cc2eba61dc5e0ef952a659f625b99f73-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region cc2eba61dc5e0ef952a659f625b99f73 2023-04-21 01:53:43,075 DEBUG [StoreOpener-cc2eba61dc5e0ef952a659f625b99f73-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/namespace/cc2eba61dc5e0ef952a659f625b99f73/info 2023-04-21 01:53:43,075 DEBUG [StoreOpener-cc2eba61dc5e0ef952a659f625b99f73-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/namespace/cc2eba61dc5e0ef952a659f625b99f73/info 2023-04-21 01:53:43,076 INFO [StoreOpener-cc2eba61dc5e0ef952a659f625b99f73-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cc2eba61dc5e0ef952a659f625b99f73 columnFamilyName info 2023-04-21 01:53:43,077 INFO [StoreOpener-cc2eba61dc5e0ef952a659f625b99f73-1] regionserver.HStore(310): Store=cc2eba61dc5e0ef952a659f625b99f73/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:53:43,079 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/namespace/cc2eba61dc5e0ef952a659f625b99f73 2023-04-21 01:53:43,080 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/namespace/cc2eba61dc5e0ef952a659f625b99f73 2023-04-21 01:53:43,086 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for cc2eba61dc5e0ef952a659f625b99f73 2023-04-21 01:53:43,092 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/namespace/cc2eba61dc5e0ef952a659f625b99f73/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-21 01:53:43,093 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened cc2eba61dc5e0ef952a659f625b99f73; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62543212, jitterRate=-0.06803351640701294}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-21 01:53:43,093 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for cc2eba61dc5e0ef952a659f625b99f73: 2023-04-21 01:53:43,096 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1682042021776.cc2eba61dc5e0ef952a659f625b99f73., pid=6, masterSystemTime=1682042023059 2023-04-21 01:53:43,102 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1682042021776.cc2eba61dc5e0ef952a659f625b99f73. 2023-04-21 01:53:43,102 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1682042021776.cc2eba61dc5e0ef952a659f625b99f73. 2023-04-21 01:53:43,104 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=cc2eba61dc5e0ef952a659f625b99f73, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:43,104 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1682042021776.cc2eba61dc5e0ef952a659f625b99f73.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682042023103"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682042023103"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682042023103"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682042023103"}]},"ts":"1682042023103"} 2023-04-21 01:53:43,114 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-21 01:53:43,114 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure cc2eba61dc5e0ef952a659f625b99f73, server=jenkins-hbase4.apache.org,40131,1682042019399 in 205 msec 2023-04-21 01:53:43,117 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-21 01:53:43,118 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=cc2eba61dc5e0ef952a659f625b99f73, ASSIGN in 369 msec 2023-04-21 01:53:43,119 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-21 01:53:43,120 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042023120"}]},"ts":"1682042023120"} 2023-04-21 01:53:43,123 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-21 01:53:43,126 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-21 01:53:43,129 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3480 sec 2023-04-21 01:53:43,190 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-21 01:53:43,193 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-21 01:53:43,193 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-21 01:53:43,234 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-21 01:53:43,256 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-21 01:53:43,263 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 37 msec 2023-04-21 01:53:43,268 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-21 01:53:43,279 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-21 01:53:43,285 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 17 msec 2023-04-21 01:53:43,297 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-21 01:53:43,299 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-21 01:53:43,300 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.821sec 2023-04-21 01:53:43,302 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-21 01:53:43,303 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-21 01:53:43,303 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-21 01:53:43,305 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35449,1682042018145-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-21 01:53:43,306 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35449,1682042018145-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-21 01:53:43,317 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-21 01:53:43,350 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x373f9c99 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:53:43,357 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6b116c72, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:53:43,370 DEBUG [hconnection-0x25e7ed72-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:53:43,385 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59220, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:53:43,397 INFO [Listener at localhost/39939] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,35449,1682042018145 2023-04-21 01:53:43,422 INFO [Listener at localhost/39939] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=205, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=444, ProcessCount=169, AvailableMemoryMB=9540 2023-04-21 01:53:43,445 DEBUG [Listener at localhost/39939] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-21 01:53:43,449 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33068, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-21 01:53:43,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-21 01:53:43,469 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-21 01:53:43,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-21 01:53:43,476 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-21 01:53:43,479 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-21 01:53:43,481 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-21 01:53:43,483 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:43,486 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8 empty. 2023-04-21 01:53:43,488 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:43,488 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-21 01:53:43,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-21 01:53:43,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-21 01:53:43,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-21 01:53:43,911 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-21 01:53:43,913 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 16b5c71a3c47da32fe9f26ec9e41dfb8, NAME => 'TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp 2023-04-21 01:53:43,931 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-21 01:53:43,931 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 16b5c71a3c47da32fe9f26ec9e41dfb8, disabling compactions & flushes 2023-04-21 01:53:43,931 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:43,931 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:43,931 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. after waiting 0 ms 2023-04-21 01:53:43,931 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:43,931 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:43,931 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:43,936 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-21 01:53:43,938 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042023938"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682042023938"}]},"ts":"1682042023938"} 2023-04-21 01:53:43,941 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-21 01:53:43,943 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-21 01:53:43,943 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042023943"}]},"ts":"1682042023943"} 2023-04-21 01:53:43,946 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-21 01:53:43,950 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=16b5c71a3c47da32fe9f26ec9e41dfb8, ASSIGN}] 2023-04-21 01:53:43,953 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=16b5c71a3c47da32fe9f26ec9e41dfb8, ASSIGN 2023-04-21 01:53:43,955 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=16b5c71a3c47da32fe9f26ec9e41dfb8, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40131,1682042019399; forceNewPlan=false, retain=false 2023-04-21 01:53:44,106 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=16b5c71a3c47da32fe9f26ec9e41dfb8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:44,107 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042024106"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682042024106"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682042024106"}]},"ts":"1682042024106"} 2023-04-21 01:53:44,110 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399}] 2023-04-21 01:53:44,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-21 01:53:44,270 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:44,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 16b5c71a3c47da32fe9f26ec9e41dfb8, NAME => 'TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.', STARTKEY => '', ENDKEY => ''} 2023-04-21 01:53:44,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:44,271 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-21 01:53:44,271 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:44,271 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:44,273 INFO [StoreOpener-16b5c71a3c47da32fe9f26ec9e41dfb8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:44,275 DEBUG [StoreOpener-16b5c71a3c47da32fe9f26ec9e41dfb8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A 2023-04-21 01:53:44,275 DEBUG [StoreOpener-16b5c71a3c47da32fe9f26ec9e41dfb8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A 2023-04-21 01:53:44,277 INFO [StoreOpener-16b5c71a3c47da32fe9f26ec9e41dfb8-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:53:44,278 INFO [StoreOpener-16b5c71a3c47da32fe9f26ec9e41dfb8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 16b5c71a3c47da32fe9f26ec9e41dfb8 columnFamilyName A 2023-04-21 01:53:44,279 INFO [StoreOpener-16b5c71a3c47da32fe9f26ec9e41dfb8-1] regionserver.HStore(310): Store=16b5c71a3c47da32fe9f26ec9e41dfb8/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:53:44,279 INFO [StoreOpener-16b5c71a3c47da32fe9f26ec9e41dfb8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:44,281 DEBUG [StoreOpener-16b5c71a3c47da32fe9f26ec9e41dfb8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B 2023-04-21 01:53:44,281 DEBUG [StoreOpener-16b5c71a3c47da32fe9f26ec9e41dfb8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B 2023-04-21 01:53:44,281 INFO [StoreOpener-16b5c71a3c47da32fe9f26ec9e41dfb8-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:53:44,282 INFO [StoreOpener-16b5c71a3c47da32fe9f26ec9e41dfb8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 16b5c71a3c47da32fe9f26ec9e41dfb8 columnFamilyName B 2023-04-21 01:53:44,282 INFO [StoreOpener-16b5c71a3c47da32fe9f26ec9e41dfb8-1] regionserver.HStore(310): Store=16b5c71a3c47da32fe9f26ec9e41dfb8/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:53:44,283 INFO [StoreOpener-16b5c71a3c47da32fe9f26ec9e41dfb8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:44,285 DEBUG [StoreOpener-16b5c71a3c47da32fe9f26ec9e41dfb8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C 2023-04-21 01:53:44,285 DEBUG [StoreOpener-16b5c71a3c47da32fe9f26ec9e41dfb8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C 2023-04-21 01:53:44,285 INFO [StoreOpener-16b5c71a3c47da32fe9f26ec9e41dfb8-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:53:44,285 INFO [StoreOpener-16b5c71a3c47da32fe9f26ec9e41dfb8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 16b5c71a3c47da32fe9f26ec9e41dfb8 columnFamilyName C 2023-04-21 01:53:44,286 INFO [StoreOpener-16b5c71a3c47da32fe9f26ec9e41dfb8-1] regionserver.HStore(310): Store=16b5c71a3c47da32fe9f26ec9e41dfb8/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:53:44,287 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:44,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:44,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:44,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-21 01:53:44,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:44,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-21 01:53:44,299 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 16b5c71a3c47da32fe9f26ec9e41dfb8; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71527790, jitterRate=0.06584712862968445}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-21 01:53:44,299 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:44,300 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., pid=11, masterSystemTime=1682042024263 2023-04-21 01:53:44,303 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:44,303 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:44,304 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=16b5c71a3c47da32fe9f26ec9e41dfb8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:44,304 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042024304"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682042024304"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682042024304"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682042024304"}]},"ts":"1682042024304"} 2023-04-21 01:53:44,311 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-21 01:53:44,311 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 in 197 msec 2023-04-21 01:53:44,314 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-21 01:53:44,315 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=16b5c71a3c47da32fe9f26ec9e41dfb8, ASSIGN in 361 msec 2023-04-21 01:53:44,315 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-21 01:53:44,316 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042024316"}]},"ts":"1682042024316"} 2023-04-21 01:53:44,318 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-21 01:53:44,321 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-21 01:53:44,324 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 853 msec 2023-04-21 01:53:44,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-21 01:53:44,614 INFO [Listener at localhost/39939] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-21 01:53:44,620 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x0b17e1c1 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:53:44,626 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@35594b1b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:53:44,629 DEBUG [Listener at localhost/39939] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:53:44,633 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59232, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:53:44,637 DEBUG [Listener at localhost/39939] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-21 01:53:44,647 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33080, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-21 01:53:44,656 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x56c8e45e to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:53:44,661 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@e16d41b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:53:44,662 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x5e91d243 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:53:44,669 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77648431, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:53:44,670 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x6038a026 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:53:44,674 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e728705, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:53:44,675 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x46192d17 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:53:44,680 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9a575b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:53:44,680 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x0dbb9ac1 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:53:44,685 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@42927198, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:53:44,686 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x6fef7a21 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:53:44,695 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fb2b82, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:53:44,696 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x2879e2e1 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:53:44,724 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3cd3abbd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:53:44,725 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x6aad2e18 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:53:44,759 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b02eeb3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:53:44,759 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x4bd6fc59 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:53:44,764 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4808c6d5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:53:44,769 DEBUG [hconnection-0x217a24de-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:53:44,772 DEBUG [hconnection-0x50724ede-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:53:44,773 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59236, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:53:44,775 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59248, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:53:44,775 DEBUG [hconnection-0x372d4e28-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:53:44,779 DEBUG [hconnection-0x6d1c3ce9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:53:44,779 DEBUG [hconnection-0x224073f8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:53:44,783 DEBUG [hconnection-0x5dfec272-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:53:44,784 DEBUG [hconnection-0x66fde656-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:53:44,784 DEBUG [hconnection-0x1516ed88-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:53:44,789 DEBUG [hconnection-0x66595227-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:53:44,795 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59256, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:53:44,796 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59262, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:53:44,798 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59276, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:53:44,799 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59272, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:53:44,799 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59292, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:53:44,800 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59306, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:53:44,801 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59312, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:53:44,811 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:44,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:44,834 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:44,834 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:44,834 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:44,834 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:44,835 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:44,835 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:44,837 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:44,837 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:44,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:44,838 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:44,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:44,838 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:44,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:44,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:44,847 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:44,857 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:44,858 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:44,861 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:44,866 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:44,866 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:44,866 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:44,870 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:44,872 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-21 01:53:44,888 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:44,889 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:44,890 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:44,890 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:44,890 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:44,890 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:44,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:44,901 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:45,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682042024959, max timestamp=1682042024986Num uniques 859; 2023-04-21 01:53:45,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-21 01:53:45,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042024965, max timestamp=1682042024986Num uniques 859; 2023-04-21 01:53:45,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:53:45,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682042024965, max timestamp=1682042024986Num uniques 859; 2023-04-21 01:53:45,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-21 01:53:45,025 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/08ce249d075d43269941240c02e68dfc 2023-04-21 01:53:45,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:45,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042085073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:45,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682042025058, max timestamp=1682042025074Num uniques 1218; 2023-04-21 01:53:45,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=55778, new segment data size=7688, suffix heap size=154640,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1218, new segment cells count=168 2023-04-21 01:53:45,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682042025058, max timestamp=1682042025074Num uniques 1018; 2023-04-21 01:53:45,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-04-21 01:53:45,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682042025058, max timestamp=1682042025076Num uniques 1009; 2023-04-21 01:53:45,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-21 01:53:45,151 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/3f82bce1e29c4cd98b5c58d26860d931 2023-04-21 01:53:45,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:45,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042085156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:45,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:45,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042085170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:45,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:45,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042085174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:45,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:45,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042085175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:45,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682042025131, max timestamp=1682042025157Num uniques 1009; 2023-04-21 01:53:45,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682042025131, max timestamp=1682042025155Num uniques 1009; 2023-04-21 01:53:45,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-21 01:53:45,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-21 01:53:45,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682042025131, max timestamp=1682042025155Num uniques 1027; 2023-04-21 01:53:45,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-21 01:53:45,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682042025294, max timestamp=1682042025302Num uniques 1009; 2023-04-21 01:53:45,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-21 01:53:45,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682042025294, max timestamp=1682042025302Num uniques 1009; 2023-04-21 01:53:45,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-21 01:53:45,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682042025294, max timestamp=1682042025300Num uniques 1009; 2023-04-21 01:53:45,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-21 01:53:45,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682042025335, max timestamp=1682042025401Num uniques 1045; 2023-04-21 01:53:45,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-21 01:53:45,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682042025335, max timestamp=1682042025401Num uniques 1045; 2023-04-21 01:53:45,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-21 01:53:45,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682042025335, max timestamp=1682042025401Num uniques 1045; 2023-04-21 01:53:45,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-21 01:53:45,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682042025428, max timestamp=1682042025449Num uniques 1054; 2023-04-21 01:53:45,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-21 01:53:45,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8424, totalHeapSize=19376, min timestamp=1682042025428, max timestamp=1682042025449Num uniques 1054; 2023-04-21 01:53:45,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48264, new segment data size=8424, suffix heap size=137584,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=184 2023-04-21 01:53:45,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682042025428, max timestamp=1682042025452Num uniques 1054; 2023-04-21 01:53:45,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-21 01:53:45,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682042025467, max timestamp=1682042025501Num uniques 1013; 2023-04-21 01:53:45,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-21 01:53:45,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682042025467, max timestamp=1682042025500Num uniques 1043; 2023-04-21 01:53:45,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47768, new segment data size=9748, suffix heap size=136440,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=213 2023-04-21 01:53:45,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682042025467, max timestamp=1682042025501Num uniques 1013; 2023-04-21 01:53:45,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=6870, suffix heap size=133320,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=150 2023-04-21 01:53:45,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682042025545, max timestamp=1682042025555Num uniques 1072; 2023-04-21 01:53:45,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682042025545, max timestamp=1682042025555Num uniques 1072; 2023-04-21 01:53:45,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-21 01:53:45,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682042025545, max timestamp=1682042025555Num uniques 1009; 2023-04-21 01:53:45,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10162, suffix heap size=132904,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=222 2023-04-21 01:53:45,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-21 01:53:45,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:45,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042085595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:45,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1682042025577, max timestamp=1682042025598Num uniques 1081; 2023-04-21 01:53:45,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-04-21 01:53:45,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12856, totalHeapSize=29464, min timestamp=1682042025577, max timestamp=1682042025598Num uniques 1081; 2023-04-21 01:53:45,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49496, new segment data size=12856, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-04-21 01:53:45,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682042025577, max timestamp=1682042025598Num uniques 1081; 2023-04-21 01:53:45,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49496, new segment data size=6870, suffix heap size=140392,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=150 2023-04-21 01:53:45,645 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/d14d111773c94bfda63e4ae73c42b364 2023-04-21 01:53:45,660 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/08ce249d075d43269941240c02e68dfc as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/08ce249d075d43269941240c02e68dfc 2023-04-21 01:53:45,690 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/08ce249d075d43269941240c02e68dfc, entries=150, sequenceid=9, filesize=11.6 K 2023-04-21 01:53:45,698 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682042025669, max timestamp=1682042025688Num uniques 1140; 2023-04-21 01:53:45,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=10990, suffix heap size=146528,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=240 2023-04-21 01:53:45,706 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10980, totalHeapSize=25200, min timestamp=1682042025669, max timestamp=1682042025692Num uniques 1009; 2023-04-21 01:53:45,706 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10980, suffix heap size=132904,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=240 2023-04-21 01:53:45,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10980, totalHeapSize=25200, min timestamp=1682042025669, max timestamp=1682042025692Num uniques 1140; 2023-04-21 01:53:45,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52200, new segment data size=10980, suffix heap size=146528,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=240 2023-04-21 01:53:45,713 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/3f82bce1e29c4cd98b5c58d26860d931 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/3f82bce1e29c4cd98b5c58d26860d931 2023-04-21 01:53:45,740 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/3f82bce1e29c4cd98b5c58d26860d931, entries=150, sequenceid=9, filesize=11.6 K 2023-04-21 01:53:45,749 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/d14d111773c94bfda63e4ae73c42b364 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/d14d111773c94bfda63e4ae73c42b364 2023-04-21 01:53:45,768 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/d14d111773c94bfda63e4ae73c42b364, entries=150, sequenceid=9, filesize=11.6 K 2023-04-21 01:53:45,772 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=53.67 KB/54960 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 900ms, sequenceid=9, compaction requested=false 2023-04-21 01:53:45,774 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-21 01:53:45,778 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:45,778 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:45,778 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:45,779 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:45,779 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:45,780 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:45,780 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:45,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:45,781 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=67.09 KB heapSize=178.17 KB 2023-04-21 01:53:45,782 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:45,782 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:45,782 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:45,782 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:45,783 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:45,783 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:45,785 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:45,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:45,785 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:45,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:45,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:45,786 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:45,786 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:45,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:45,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:45,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:45,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:45,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:45,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a7e9652[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:45,790 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:45,790 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:45,798 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:45,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:45,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:45,798 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:45,798 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:45,798 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:45,816 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:45,816 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:45,816 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:45,816 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:45,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:45,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:45,817 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:45,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:45,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:45,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:45,817 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:45,820 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:45,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:45,822 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:45,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:45,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:45,830 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:45,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:45,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:45,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:45,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:45,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:45,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:45,849 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:45,849 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:45,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:45,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:45,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:45,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:45,860 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:45,860 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:45,860 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:45,860 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:45,860 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:45,860 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:45,860 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:45,860 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:45,864 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:45,864 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:45,863 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:45,868 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:45,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:45,932 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/9e710ac33917431fbd44470724b9dd43 2023-04-21 01:53:45,969 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:45,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:45,982 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:46,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042085954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042085956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042085967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042085972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042086003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:46,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:46,064 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:46,065 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:46,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:46,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:46,067 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/17f1d4a52e2047e3a7191a0ca78d4b70 2023-04-21 01:53:46,067 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,068 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:46,069 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:46,069 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:46,070 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,070 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:46,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:46,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:46,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:46,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:46,077 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:46,077 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:46,078 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:46,080 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:46,080 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:46,080 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:46,080 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:46,080 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:46,080 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:46,081 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:46,081 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:46,081 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,081 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:46,081 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:46,087 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,087 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:46,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:46,088 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,088 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:46,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:46,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:46,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:46,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5faddf99[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:46,110 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:46,110 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,112 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,112 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:46,112 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:46,112 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:46,113 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,113 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:46,115 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:46,115 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:46,115 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:46,116 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:46,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:46,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:46,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:46,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:46,160 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:46,160 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:46,161 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:46,161 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:46,161 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:46,170 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,170 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:46,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:46,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,170 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:46,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:46,180 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:46,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:46,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:46,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,183 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:46,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,198 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:46,198 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:46,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:46,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:46,198 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,198 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:46,198 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:46,199 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,199 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:46,199 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:46,199 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:46,199 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:46,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:46,205 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:46,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:46,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042086144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,199 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,206 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,205 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:46,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042086145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042086146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042086157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042086157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,306 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:46,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:46,322 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:46,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:46,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:46,330 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:46,330 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:46,332 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:46,332 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:46,335 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,335 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:46,341 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:46,341 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:46,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:46,342 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,342 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:46,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:46,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:46,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:46,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:46,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:46,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:46,348 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:46,348 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:46,352 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:46,356 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:46,356 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:46,356 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:46,356 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:46,356 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:46,356 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:46,357 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:46,357 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:46,358 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,358 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:46,358 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,362 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:46,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:46,363 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,363 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:46,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:46,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:46,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:46,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:46,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1760c9be[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:46,366 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:46,366 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,370 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,370 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:46,370 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:46,370 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:46,372 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,372 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:46,396 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:46,397 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:46,397 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:46,397 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:46,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:46,403 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:46,403 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:46,403 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:46,403 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:46,403 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:46,405 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,405 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:46,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,405 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:46,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:46,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:46,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:46,408 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:46,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,415 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042086410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042086410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042086411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042086411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,436 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:46,436 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:46,437 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,437 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:46,437 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:46,437 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:46,437 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:46,438 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:46,438 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,438 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,438 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,439 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:46,439 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:46,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:46,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:46,439 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:46,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042086413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:46,539 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:46,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:46,547 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:46,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:46,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:46,557 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:46,557 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:46,558 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:46,559 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:46,560 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,560 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:46,563 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:46,563 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:46,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:46,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:46,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,565 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,565 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:46,566 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:46,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:46,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:46,580 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/2d8420ee76ac438aa3b8a6fdffe5d901 2023-04-21 01:53:46,580 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:46,580 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:46,591 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:46,595 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:46,595 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:46,595 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:46,595 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:46,596 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:46,596 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:46,599 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/9e710ac33917431fbd44470724b9dd43 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/9e710ac33917431fbd44470724b9dd43 2023-04-21 01:53:46,603 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:46,606 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:46,606 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,606 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:46,606 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:46,609 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,609 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:46,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:46,609 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,609 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:46,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:46,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:46,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:46,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ed46a78[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:46,623 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:46,623 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,624 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,625 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,625 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:46,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,625 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:46,625 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:46,625 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:46,629 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:46,633 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:46,634 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:46,634 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:46,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:46,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,639 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/9e710ac33917431fbd44470724b9dd43, entries=300, sequenceid=189, filesize=18.8 K 2023-04-21 01:53:46,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:46,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:46,642 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:46,642 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:46,642 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:46,642 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:46,642 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:46,642 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/17f1d4a52e2047e3a7191a0ca78d4b70 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/17f1d4a52e2047e3a7191a0ca78d4b70 2023-04-21 01:53:46,644 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,644 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,644 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:46,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:46,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:46,646 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,648 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:46,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,660 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:46,660 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:46,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:46,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:46,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:46,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:46,661 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/17f1d4a52e2047e3a7191a0ca78d4b70, entries=150, sequenceid=189, filesize=11.8 K 2023-04-21 01:53:46,664 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,664 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:46,664 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:46,664 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:46,664 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:46,664 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:46,664 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,664 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,664 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,664 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,665 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:46,665 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:46,671 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/2d8420ee76ac438aa3b8a6fdffe5d901 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/2d8420ee76ac438aa3b8a6fdffe5d901 2023-04-21 01:53:46,684 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/2d8420ee76ac438aa3b8a6fdffe5d901, entries=150, sequenceid=189, filesize=11.8 K 2023-04-21 01:53:46,686 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~178.13 KB/182400, currentSize=114.05 KB/116790 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 905ms, sequenceid=189, compaction requested=false 2023-04-21 01:53:46,687 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:46,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042025951, max timestamp=1682042026720Num uniques 859; 2023-04-21 01:53:46,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042025951, max timestamp=1682042026720Num uniques 859; 2023-04-21 01:53:46,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:53:46,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:53:46,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042025951, max timestamp=1682042026720Num uniques 859; 2023-04-21 01:53:46,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:53:46,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:46,750 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=53.67 KB heapSize=145.92 KB 2023-04-21 01:53:46,753 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:46,754 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:46,754 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:46,754 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:46,754 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:46,754 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:46,765 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:46,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:46,772 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:46,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042086805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:46,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:46,812 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:46,812 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:46,814 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682042026782, max timestamp=1682042026805Num uniques 859; 2023-04-21 01:53:46,814 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682042026782, max timestamp=1682042026805Num uniques 859; 2023-04-21 01:53:46,814 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-21 01:53:46,816 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682042026782, max timestamp=1682042026805Num uniques 859; 2023-04-21 01:53:46,816 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-21 01:53:46,814 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-21 01:53:46,829 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:46,829 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:46,836 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,836 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:46,838 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:46,838 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:46,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:46,839 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:46,839 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:46,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:46,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:46,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:46,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042086856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042086858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:46,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042086860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:46,872 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:46,874 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:46,875 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:46,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042086865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:46,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:46,882 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:46,883 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:46,883 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:46,883 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:46,888 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:46,888 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:46,888 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,888 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:46,888 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:46,891 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,891 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:46,892 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,892 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:46,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:46,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:46,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:46,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3026b1f6[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:46,894 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:46,894 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,896 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,896 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:46,897 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:46,897 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,897 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:46,897 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:46,899 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:46,899 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:46,900 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:46,912 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:46,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:46,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:46,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:46,918 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:46,918 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:46,918 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:46,918 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:46,918 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:46,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042086914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,920 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,920 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:46,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:46,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,921 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:46,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:46,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:46,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:46,922 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,924 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:46,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,932 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,932 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:46,932 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:46,932 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,932 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:46,932 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:46,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:46,940 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:46,940 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:46,932 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:46,932 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:46,940 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:46,940 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:46,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:46,941 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:46,941 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:46,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:46,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:46,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042086964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042086965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,982 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042086977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:46,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:46,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042086977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,041 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:47,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:47,047 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:47,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:47,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:47,060 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:47,061 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:47,062 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:47,062 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:47,063 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,063 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:47,065 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,065 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:47,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:47,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:47,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:47,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:47,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:47,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:47,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:47,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:47,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:47,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:47,069 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:47,080 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:47,080 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:47,080 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:47,080 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:47,080 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:47,080 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:47,082 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:47,082 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:47,082 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,082 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:47,082 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:47,085 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,085 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:47,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:47,085 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,085 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:47,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:47,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:47,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:47,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:47,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:47,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c6e17c[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:47,090 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:47,090 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,093 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,094 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:47,094 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:47,094 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:47,094 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,094 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:47,096 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,096 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:47,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:47,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:47,096 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:47,096 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:47,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:47,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:47,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:47,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:47,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:47,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:47,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:47,100 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:47,101 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:47,101 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:47,101 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:47,101 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:47,103 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,104 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:47,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,104 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:47,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,109 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,112 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:47,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:47,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,116 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:47,116 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:47,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:47,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:47,116 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,116 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:47,116 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:47,116 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,116 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,116 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,116 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,116 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:47,117 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:47,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:47,116 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:47,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:47,140 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:47,146 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:47,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:47,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042087131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:47,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042087168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:47,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042087170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,181 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-21 01:53:47,188 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:47,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042087186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:47,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042087187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,246 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:47,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:47,275 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:47,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:47,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:47,282 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:47,282 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:47,285 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:47,285 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:47,286 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,286 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:47,288 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,289 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,289 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:47,289 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:47,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:47,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:47,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:47,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:47,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:47,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:47,311 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:47,311 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:47,324 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:47,324 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:47,324 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:47,324 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/dfc9ddd793b641ffb0a6bf196b91d94e 2023-04-21 01:53:47,324 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:47,325 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:47,325 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:47,325 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:47,362 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:47,362 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:47,362 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,362 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:47,362 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:47,366 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,366 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:47,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:47,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:47,367 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,367 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:47,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:47,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:47,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@107af5c3[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:47,370 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:47,370 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,372 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,372 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:47,372 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:47,372 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:47,372 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,372 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:47,393 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:47,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:47,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:47,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:47,419 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-21 01:53:47,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:47,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:47,422 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-21 01:53:47,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:47,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:47,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:47,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:47,425 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:47,425 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:47,425 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:47,425 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:47,425 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:47,428 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,428 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,428 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:47,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:47,429 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:47,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:47,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:47,432 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:47,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:47,460 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:47,460 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:47,461 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:47,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:47,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:47,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:47,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:47,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:47,466 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:47,466 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:47,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:47,467 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/8cbcaf0f89b74b91a066d395b5c41bb2 2023-04-21 01:53:47,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:47,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:47,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042087459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:47,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042087476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:47,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042087482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:47,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042087492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:47,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042087493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,570 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:47,580 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/8333f0b9c3c04d4ca22ee1d56f706828 2023-04-21 01:53:47,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:47,591 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:47,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:47,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:47,593 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:47,593 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:47,594 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:47,594 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:47,597 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,597 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:47,599 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/dfc9ddd793b641ffb0a6bf196b91d94e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/dfc9ddd793b641ffb0a6bf196b91d94e 2023-04-21 01:53:47,600 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,600 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:47,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:47,601 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,601 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:47,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:47,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:47,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:47,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:47,604 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:47,604 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:47,605 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:47,606 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:47,606 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:47,606 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:47,606 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:47,606 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:47,606 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:47,606 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:47,606 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:47,606 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,606 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:47,606 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:47,608 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:47,610 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,610 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,610 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:47,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:47,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dd0ca1e[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:47,614 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:47,614 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,614 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/dfc9ddd793b641ffb0a6bf196b91d94e, entries=150, sequenceid=215, filesize=11.8 K 2023-04-21 01:53:47,617 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,617 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:47,617 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:47,618 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,618 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:47,618 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:47,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,619 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:47,619 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:47,620 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:47,621 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/8cbcaf0f89b74b91a066d395b5c41bb2 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/8cbcaf0f89b74b91a066d395b5c41bb2 2023-04-21 01:53:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:47,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:47,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:47,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:47,630 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:47,630 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:47,630 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:47,630 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:47,630 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:47,640 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,640 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,640 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:47,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:47,641 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:47,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:47,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:47,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:47,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:47,646 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,667 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,667 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:47,667 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:47,667 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:47,667 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:47,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:47,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:47,667 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:47,692 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:47,692 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,692 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:47,693 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:47,693 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:47,680 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/8cbcaf0f89b74b91a066d395b5c41bb2, entries=150, sequenceid=215, filesize=11.8 K 2023-04-21 01:53:47,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:47,692 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:47,697 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:47,697 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,701 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/8333f0b9c3c04d4ca22ee1d56f706828 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/8333f0b9c3c04d4ca22ee1d56f706828 2023-04-21 01:53:47,725 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/8333f0b9c3c04d4ca22ee1d56f706828, entries=150, sequenceid=215, filesize=11.8 K 2023-04-21 01:53:47,729 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~165.80 KB/169776, currentSize=132.99 KB/136185 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 979ms, sequenceid=215, compaction requested=true 2023-04-21 01:53:47,730 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:47,738 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:47,738 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:53:47,739 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:53:47,739 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:53:47,741 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:53:47,746 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35958 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:53:47,748 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/B is initiating minor compaction (all files) 2023-04-21 01:53:47,749 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/B in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:47,749 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/3f82bce1e29c4cd98b5c58d26860d931, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/17f1d4a52e2047e3a7191a0ca78d4b70, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/8cbcaf0f89b74b91a066d395b5c41bb2] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=35.1 K 2023-04-21 01:53:47,750 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 43128 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:53:47,750 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/A is initiating minor compaction (all files) 2023-04-21 01:53:47,750 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/A in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:47,750 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/08ce249d075d43269941240c02e68dfc, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/9e710ac33917431fbd44470724b9dd43, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/dfc9ddd793b641ffb0a6bf196b91d94e] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=42.1 K 2023-04-21 01:53:47,751 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 3f82bce1e29c4cd98b5c58d26860d931, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1682042024836 2023-04-21 01:53:47,753 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 08ce249d075d43269941240c02e68dfc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1682042024836 2023-04-21 01:53:47,756 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 9e710ac33917431fbd44470724b9dd43, keycount=300, bloomtype=ROW, size=18.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1682042025678 2023-04-21 01:53:47,757 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 17f1d4a52e2047e3a7191a0ca78d4b70, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1682042025692 2023-04-21 01:53:47,758 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 8cbcaf0f89b74b91a066d395b5c41bb2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1682042026728 2023-04-21 01:53:47,758 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting dfc9ddd793b641ffb0a6bf196b91d94e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1682042026728 2023-04-21 01:53:47,793 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:47,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:47,800 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:47,816 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#B#compaction#9 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:47,832 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#A#compaction#10 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:47,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:47,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:47,866 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:47,866 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:47,867 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:47,868 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:47,876 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,876 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:47,878 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:47,878 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:47,879 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:47,879 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:47,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:47,880 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:47,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:47,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:47,881 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:47,881 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:47,900 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:47,916 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:47,916 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:47,916 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:47,916 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:47,916 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=132.99 KB heapSize=382.24 KB 2023-04-21 01:53:47,928 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:47,929 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:47,929 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:47,929 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:47,929 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:47,929 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:47,988 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/b5cd117cbbd84aa6a0ba94ab0c128ba8 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b5cd117cbbd84aa6a0ba94ab0c128ba8 2023-04-21 01:53:47,994 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/cb2aa42de24747de9fd3cc3473ec63ee as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/cb2aa42de24747de9fd3cc3473ec63ee 2023-04-21 01:53:48,011 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:48,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:48,030 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/B of 16b5c71a3c47da32fe9f26ec9e41dfb8 into cb2aa42de24747de9fd3cc3473ec63ee(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:48,031 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:48,031 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/B, priority=13, startTime=1682042027738; duration=0sec 2023-04-21 01:53:48,032 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/A of 16b5c71a3c47da32fe9f26ec9e41dfb8 into b5cd117cbbd84aa6a0ba94ab0c128ba8(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:48,032 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:48,032 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/A, priority=13, startTime=1682042027733; duration=0sec 2023-04-21 01:53:48,032 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:53:48,033 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:53:48,033 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:48,041 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35958 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:53:48,041 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/C is initiating minor compaction (all files) 2023-04-21 01:53:48,042 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/C in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:48,042 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/d14d111773c94bfda63e4ae73c42b364, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/2d8420ee76ac438aa3b8a6fdffe5d901, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/8333f0b9c3c04d4ca22ee1d56f706828] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=35.1 K 2023-04-21 01:53:48,043 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting d14d111773c94bfda63e4ae73c42b364, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1682042024836 2023-04-21 01:53:48,044 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 2d8420ee76ac438aa3b8a6fdffe5d901, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1682042025692 2023-04-21 01:53:48,045 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 8333f0b9c3c04d4ca22ee1d56f706828, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1682042026728 2023-04-21 01:53:48,075 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#C#compaction#12 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:48,088 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:48,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:48,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042088042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:48,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042088039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042088040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:48,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042088042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:48,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042088088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:48,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042088204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,211 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:48,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042088204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:48,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042088206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:48,216 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:48,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042088207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042088206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,302 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-21 01:53:48,302 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-21 01:53:48,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:48,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042088410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:48,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042088416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:48,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042088418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:48,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042088418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:48,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042088419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,442 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/0807cca5c9134179a1a71f07f50da02e 2023-04-21 01:53:48,520 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/c3fe54704263476cb315207d4e23c1c6 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/c3fe54704263476cb315207d4e23c1c6 2023-04-21 01:53:48,532 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/C of 16b5c71a3c47da32fe9f26ec9e41dfb8 into c3fe54704263476cb315207d4e23c1c6(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:48,532 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:48,532 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/C, priority=13, startTime=1682042027739; duration=0sec 2023-04-21 01:53:48,532 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:48,723 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:48,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042088718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:48,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042088721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,751 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:48,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042088750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:48,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042088752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:48,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042088752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,888 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/30e2da97fac6451a9247350c25b12b6a 2023-04-21 01:53:48,918 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/c500a9a9d8fa40f089d191b93770dcc1 2023-04-21 01:53:48,928 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/0807cca5c9134179a1a71f07f50da02e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/0807cca5c9134179a1a71f07f50da02e 2023-04-21 01:53:48,936 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/0807cca5c9134179a1a71f07f50da02e, entries=150, sequenceid=252, filesize=11.8 K 2023-04-21 01:53:48,943 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/30e2da97fac6451a9247350c25b12b6a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/30e2da97fac6451a9247350c25b12b6a 2023-04-21 01:53:48,965 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/30e2da97fac6451a9247350c25b12b6a, entries=150, sequenceid=252, filesize=11.8 K 2023-04-21 01:53:48,970 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/c500a9a9d8fa40f089d191b93770dcc1 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/c500a9a9d8fa40f089d191b93770dcc1 2023-04-21 01:53:48,981 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/c500a9a9d8fa40f089d191b93770dcc1, entries=150, sequenceid=252, filesize=11.8 K 2023-04-21 01:53:48,987 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~132.99 KB/136185, heapSize ~382.20 KB/391368, currentSize=46.96 KB/48090 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 1071ms, sequenceid=252, compaction requested=false 2023-04-21 01:53:48,987 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:48,987 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:48,987 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:48,988 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:48,988 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,988 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:48,988 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:48,990 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,990 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:48,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:48,992 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:48,992 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:48,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:48,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:48,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38fc7553[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:48,995 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:48,995 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:48,996 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:48,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:48,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:48,996 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:48,996 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:48,996 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:48,997 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,997 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:48,998 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:48,998 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:48,998 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:48,998 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:49,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:49,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:49,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:49,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:49,017 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:49,017 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:49,017 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:49,017 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:49,017 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:49,020 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:49,020 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:49,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:49,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:49,021 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,022 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:49,024 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:49,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:49,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:49,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:49,031 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,031 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:49,031 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:49,031 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,031 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:49,031 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:49,031 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:49,031 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:49,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:49,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:49,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:49,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:49,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:49,031 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:49,035 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:49,035 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:49,037 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:49,037 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:49,037 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:49,137 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:49,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:49,141 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:49,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:49,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:49,143 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:49,143 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:49,144 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:49,144 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:49,145 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:49,145 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:49,149 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,149 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:49,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:49,150 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:49,150 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:49,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:49,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:49,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:49,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:49,155 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:49,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:49,155 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:49,155 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:49,158 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:49,158 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:49,158 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:49,158 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:49,158 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:49,159 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-21 01:53:49,159 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:49,159 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:49,159 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:49,159 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:49,159 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:49,159 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:49,198 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/44441447fd5146dcbbcb9448d88f959e 2023-04-21 01:53:49,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:49,230 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:49,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682042029264, max timestamp=1682042029300Num uniques 859; 2023-04-21 01:53:49,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682042029264, max timestamp=1682042029300Num uniques 859; 2023-04-21 01:53:49,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682042029264, max timestamp=1682042029300Num uniques 859; 2023-04-21 01:53:49,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-21 01:53:49,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-21 01:53:49,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-21 01:53:49,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:49,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042089324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:49,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042089324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:49,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:49,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042089325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042089326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:49,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042089326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:49,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042089429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:49,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042089437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:49,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042089438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:49,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042089438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:49,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042089438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:49,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042089633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:49,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042089642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:49,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042089642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:49,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042089642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:49,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042089642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,651 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/de11981e03ea4ebc9d406480c7578199 2023-04-21 01:53:49,679 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/408896e8c85a49a5a55e1d2fad7e42d9 2023-04-21 01:53:49,688 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/44441447fd5146dcbbcb9448d88f959e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/44441447fd5146dcbbcb9448d88f959e 2023-04-21 01:53:49,697 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/44441447fd5146dcbbcb9448d88f959e, entries=150, sequenceid=265, filesize=11.9 K 2023-04-21 01:53:49,699 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/de11981e03ea4ebc9d406480c7578199 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/de11981e03ea4ebc9d406480c7578199 2023-04-21 01:53:49,714 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/de11981e03ea4ebc9d406480c7578199, entries=150, sequenceid=265, filesize=11.9 K 2023-04-21 01:53:49,716 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/408896e8c85a49a5a55e1d2fad7e42d9 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/408896e8c85a49a5a55e1d2fad7e42d9 2023-04-21 01:53:49,746 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/408896e8c85a49a5a55e1d2fad7e42d9, entries=150, sequenceid=265, filesize=11.9 K 2023-04-21 01:53:49,749 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 589ms, sequenceid=265, compaction requested=true 2023-04-21 01:53:49,749 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:49,749 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:49,749 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:49,753 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:49,753 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,753 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:49,753 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:49,755 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:49,755 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:49,755 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:49,755 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:49,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:49,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e07be31[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:49,759 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:49,759 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:49,766 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:49,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:49,766 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:49,766 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:49,766 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:49,768 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:49,768 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:49,769 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:49,778 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,778 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:49,779 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:49,782 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:49,782 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:49,782 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:49,782 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:49,782 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:49,786 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:49,786 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:49,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:49,786 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:49,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:49,788 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,791 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,797 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:49,797 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:49,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:49,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:49,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:49,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:49,798 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,798 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:49,799 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:49,799 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,799 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:49,799 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:49,799 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:49,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:49,799 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:49,799 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:49,799 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:49,799 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:49,800 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:49,900 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:49,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:49,904 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:49,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:49,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:49,906 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:49,906 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:49,909 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:49,909 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:49,910 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:49,910 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:49,911 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,912 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,912 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:49,912 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:49,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:49,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:49,914 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:49,914 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:49,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:49,914 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:49,914 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:49,916 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:49,916 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:49,916 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:49,916 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:49,916 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:49,916 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-21 01:53:49,917 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:49,917 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:49,917 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:49,917 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:49,917 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:49,917 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:49,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:49,939 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:49,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:49,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042089958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:49,967 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/b905972c1fbd49a58a426eb0f7c29905 2023-04-21 01:53:49,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042089959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:49,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042089961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:49,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042089962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:49,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:49,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042089964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,010 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/a59cca98e04641ad955f732fbe59657c 2023-04-21 01:53:50,041 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/eecbfb279e6644898a87caa604a1c345 2023-04-21 01:53:50,051 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/b905972c1fbd49a58a426eb0f7c29905 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b905972c1fbd49a58a426eb0f7c29905 2023-04-21 01:53:50,061 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b905972c1fbd49a58a426eb0f7c29905, entries=150, sequenceid=301, filesize=11.9 K 2023-04-21 01:53:50,064 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/a59cca98e04641ad955f732fbe59657c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/a59cca98e04641ad955f732fbe59657c 2023-04-21 01:53:50,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:50,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042090069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:50,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042090070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:50,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042090070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:50,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042090070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:50,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042090071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,079 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/a59cca98e04641ad955f732fbe59657c, entries=150, sequenceid=301, filesize=11.9 K 2023-04-21 01:53:50,082 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/eecbfb279e6644898a87caa604a1c345 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/eecbfb279e6644898a87caa604a1c345 2023-04-21 01:53:50,094 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/eecbfb279e6644898a87caa604a1c345, entries=150, sequenceid=301, filesize=11.9 K 2023-04-21 01:53:50,097 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=46.96 KB/48090 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 180ms, sequenceid=301, compaction requested=true 2023-04-21 01:53:50,097 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:50,097 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:50,097 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:50,098 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:50,098 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,098 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:50,098 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:50,100 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,100 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:50,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:50,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:50,101 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:50,101 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:50,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:50,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:50,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:50,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@499486df[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:50,104 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:50,104 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:50,107 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:50,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:50,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:50,107 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:50,107 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:50,107 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:50,108 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,108 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:50,110 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:50,110 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:50,110 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:50,112 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:50,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:50,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:50,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:50,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:50,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:50,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:50,123 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:50,123 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:50,123 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:50,123 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:50,123 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:50,125 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:50,125 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:50,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:50,126 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:50,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:50,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:50,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:50,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:50,128 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,130 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:50,135 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:50,141 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,141 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:50,141 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:50,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:50,141 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:50,141 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:50,142 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,142 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:50,142 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:50,142 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:50,144 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:50,144 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:50,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:50,145 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:50,145 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:50,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:50,245 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:50,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:50,249 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:50,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:50,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:50,254 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:50,254 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:50,257 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:50,257 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:50,257 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:50,258 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:50,259 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:50,259 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:50,260 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,260 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:50,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:50,263 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:50,263 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:50,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:50,263 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:50,263 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:50,266 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:50,266 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:50,266 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:50,272 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:50,272 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:50,272 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-21 01:53:50,273 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:50,273 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:50,273 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:50,273 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:50,273 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:50,273 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:50,284 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:50,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:50,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:50,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042090316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682042030305, max timestamp=1682042030317Num uniques 859; 2023-04-21 01:53:50,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682042030305, max timestamp=1682042030317Num uniques 859; 2023-04-21 01:53:50,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-21 01:53:50,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-21 01:53:50,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682042030305, max timestamp=1682042030317Num uniques 859; 2023-04-21 01:53:50,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-21 01:53:50,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1682042030354, max timestamp=1682042030368Num uniques 1009; 2023-04-21 01:53:50,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=14558, suffix heap size=132904,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=318 2023-04-21 01:53:50,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1682042030354, max timestamp=1682042030368Num uniques 1009; 2023-04-21 01:53:50,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1682042030354, max timestamp=1682042030368Num uniques 1009; 2023-04-21 01:53:50,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=14558, suffix heap size=132904,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=318 2023-04-21 01:53:50,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=14558, suffix heap size=132904,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=318 2023-04-21 01:53:50,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:50,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042090399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:50,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042090400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:50,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:50,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042090401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042090402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,428 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:50,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042090426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:50,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042090503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:50,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042090508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:50,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042090508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:50,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042090511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:50,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042090631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:50,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042090714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:50,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042090717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:50,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:50,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042090721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042090721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,767 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/4a48a64bba844b598c56189250f3569f 2023-04-21 01:53:50,819 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/7a749b8b1c274a979edfab794e5df3dd 2023-04-21 01:53:50,853 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/6d4e5f6783174912a7fe06b88da34d9b 2023-04-21 01:53:50,862 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/4a48a64bba844b598c56189250f3569f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/4a48a64bba844b598c56189250f3569f 2023-04-21 01:53:50,872 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/4a48a64bba844b598c56189250f3569f, entries=150, sequenceid=311, filesize=11.9 K 2023-04-21 01:53:50,875 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/7a749b8b1c274a979edfab794e5df3dd as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/7a749b8b1c274a979edfab794e5df3dd 2023-04-21 01:53:50,884 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/7a749b8b1c274a979edfab794e5df3dd, entries=150, sequenceid=311, filesize=11.9 K 2023-04-21 01:53:50,887 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/6d4e5f6783174912a7fe06b88da34d9b as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/6d4e5f6783174912a7fe06b88da34d9b 2023-04-21 01:53:50,896 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/6d4e5f6783174912a7fe06b88da34d9b, entries=150, sequenceid=311, filesize=11.9 K 2023-04-21 01:53:50,899 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 626ms, sequenceid=311, compaction requested=true 2023-04-21 01:53:50,899 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:50,899 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:50,899 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:50,905 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:50,905 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,905 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:50,905 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:50,909 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:50,909 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:50,910 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:50,910 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:50,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:50,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:50,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@722f35c9[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:50,913 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:50,913 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:50,914 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:50,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:50,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:50,915 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,915 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:50,917 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:50,917 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,917 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:50,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:50,917 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:50,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:50,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:50,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:50,922 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:50,922 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:50,922 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:50,922 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:50,922 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:50,924 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:50,924 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:50,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:50,924 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:50,925 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,927 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,933 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,933 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:50,933 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:50,933 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:50,933 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:50,933 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:50,933 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:50,933 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:50,933 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:50,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:50,933 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:50,933 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:50,933 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:50,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:50,935 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:50,935 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:50,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:50,941 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=147.60 KB heapSize=409.92 KB 2023-04-21 01:53:50,941 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:50,941 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:50,941 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:50,941 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:50,941 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:50,941 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:50,970 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/adb5a5aa3a2f4e89b034fd704b422996 2023-04-21 01:53:50,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:50,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042090985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:51,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:51,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042091025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042091025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:51,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042091026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:51,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042091027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,035 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:51,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:51,039 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:51,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:51,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:51,062 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:51,062 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:51,064 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:51,064 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:51,064 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,064 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,067 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:51,067 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:51,067 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,068 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:51,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:51,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:51,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:51,070 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:51,070 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:51,071 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:51,072 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:51,072 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:51,072 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:51,072 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:51,072 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:51,072 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:51,072 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:51,072 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:51,072 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,073 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:51,073 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:51,075 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,075 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:51,076 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:51,076 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:51,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:51,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29a28032[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:51,079 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:51,079 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,087 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,087 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:51,087 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:51,087 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:51,087 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,087 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:51,090 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:51,090 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:51,090 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:51,091 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:51,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042091093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:51,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:51,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:51,099 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:51,099 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:51,099 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:51,099 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:51,099 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:51,101 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,101 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:51,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,102 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:51,103 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:51,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:51,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,107 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,115 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,115 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,115 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,115 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,115 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,115 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,115 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,115 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:51,115 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:51,115 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:51,115 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:51,116 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:51,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:51,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:51,123 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:51,123 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:51,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:51,223 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:51,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:51,226 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:51,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:51,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:51,228 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:51,228 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:51,229 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:51,229 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:51,230 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,230 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,231 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:51,231 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:51,232 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,232 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:51,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:51,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:51,233 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:51,233 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:51,234 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:51,234 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:51,235 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:51,235 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:51,235 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:51,235 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:51,236 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,236 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:51,237 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,237 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:51,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:51,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a675501[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:51,239 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:51,239 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,240 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,240 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:51,240 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:51,240 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:51,240 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,240 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:51,243 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:51,243 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,243 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:51,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:51,243 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:51,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:51,247 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:51,247 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:51,247 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:51,247 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:51,247 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:51,248 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,248 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:51,249 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,249 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:51,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:51,251 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,255 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:51,255 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:51,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:51,255 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:51,256 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,256 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,256 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,256 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,256 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,256 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:51,256 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:51,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:51,256 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:51,257 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:51,258 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:51,258 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:51,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:51,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042091296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,358 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:51,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:51,361 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:51,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:51,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:51,363 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:51,363 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:51,365 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:51,365 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:51,365 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,365 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,367 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,367 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:51,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:51,367 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,367 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,368 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:51,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:51,369 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:51,369 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:51,369 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:51,370 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:51,370 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:51,370 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:51,370 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:51,370 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:51,370 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:51,370 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:51,370 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:51,370 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:51,370 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,370 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:51,371 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:51,373 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,373 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:51,374 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,374 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:51,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:51,376 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:51,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77289aec[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:51,376 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,380 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,380 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:51,380 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:51,380 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,380 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:51,380 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:51,382 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:51,382 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,382 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:51,382 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:51,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:51,386 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:51,386 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:51,387 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:51,387 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:51,387 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:51,389 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,389 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,389 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:51,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:51,391 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:51,391 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/b7c969d50f884cf4bf34a136a6d8eb73 2023-04-21 01:53:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:51,398 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,403 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,403 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,403 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,403 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:51,403 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:51,404 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:51,403 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,403 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:51,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:51,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:51,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:51,404 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:51,404 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,405 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,405 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,405 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:51,405 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:51,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:51,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:51,505 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:51,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:51,509 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:51,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:51,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:51,510 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:51,511 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:51,512 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:51,512 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:51,512 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,512 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,514 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:51,514 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:51,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:51,514 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,514 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:51,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:51,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:51,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:51,516 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:51,516 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:51,519 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:51,519 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:51,520 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:51,520 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:51,520 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:51,520 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:51,521 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,521 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:51,521 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:51,522 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,522 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:51,522 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,523 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:51,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:51,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ca268bd[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:51,526 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:51,526 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,542 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,543 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:51,543 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:51,543 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:51,543 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,543 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:51,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:51,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042091544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:51,545 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042091544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:51,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:51,545 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:51,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042091544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:51,546 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:51,546 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:51,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:51,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:51,550 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:51,550 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:51,550 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:51,550 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:51,550 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:51,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:51,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042091545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,552 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,552 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:51,552 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:51,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:51,553 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,554 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,557 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:51,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:51,557 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:51,558 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,558 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,558 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,558 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,558 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,558 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:51,558 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:51,558 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:51,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:51,558 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:51,558 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:51,560 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:51,564 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:51,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:51,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59276 deadline: 1682042091603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,664 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:51,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:51,667 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:51,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:51,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:51,669 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:51,669 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:51,670 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:51,670 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:51,670 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,670 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,673 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:51,673 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:51,673 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,674 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:51,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:51,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:51,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:51,675 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:51,675 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:51,678 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:51,678 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:51,678 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:51,678 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:51,679 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:51,679 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:51,679 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:51,679 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:51,679 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,679 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:51,679 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:51,680 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,680 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:51,681 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,681 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:51,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:51,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41bcd40b[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:51,682 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:51,683 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,684 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,684 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:51,684 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:51,684 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:51,684 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,685 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:51,686 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:51,686 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:51,686 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:51,686 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:51,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:51,690 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:51,690 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:51,690 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:51,690 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:51,690 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:51,697 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,697 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:51,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:51,697 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:51,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,697 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:51,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,699 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,702 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,702 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,702 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,702 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,702 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,703 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,703 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,703 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:51,703 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:51,703 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:51,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:51,704 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:51,704 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:51,704 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:51,704 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:51,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:51,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:51,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:51,804 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:51,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:51,807 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:51,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:51,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:51,809 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:51,809 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:51,810 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:51,810 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:51,810 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,811 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,812 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,812 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:51,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:51,812 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,812 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:51,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:51,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:51,814 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:51,814 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:51,815 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:51,815 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:51,815 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:51,815 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:51,815 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:51,815 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:51,815 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:51,815 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:51,815 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:51,815 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,815 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:51,815 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:51,817 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,817 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:51,817 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,817 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:51,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:51,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:51,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:51,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61d35e60[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:51,819 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:51,819 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,822 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,822 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,822 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:51,822 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:51,822 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:51,822 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:51,824 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:51,824 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,824 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:51,824 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:51,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:51,827 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:51,827 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:51,827 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:51,827 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:51,827 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:51,831 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/c0d44205feb04bc58e7d3476dac1014c 2023-04-21 01:53:51,831 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,831 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,831 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:51,832 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:51,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,834 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,838 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:51,838 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:51,838 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:51,839 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:51,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:51,839 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:51,839 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:51,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,839 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:51,839 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:51,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:51,841 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:51,842 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/adb5a5aa3a2f4e89b034fd704b422996 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/adb5a5aa3a2f4e89b034fd704b422996 2023-04-21 01:53:51,852 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/adb5a5aa3a2f4e89b034fd704b422996, entries=200, sequenceid=364, filesize=14.3 K 2023-04-21 01:53:51,854 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/b7c969d50f884cf4bf34a136a6d8eb73 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/b7c969d50f884cf4bf34a136a6d8eb73 2023-04-21 01:53:51,881 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/b7c969d50f884cf4bf34a136a6d8eb73, entries=150, sequenceid=364, filesize=11.9 K 2023-04-21 01:53:51,884 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/c0d44205feb04bc58e7d3476dac1014c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/c0d44205feb04bc58e7d3476dac1014c 2023-04-21 01:53:51,892 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/c0d44205feb04bc58e7d3476dac1014c, entries=150, sequenceid=364, filesize=11.9 K 2023-04-21 01:53:51,894 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~409.88 KB/419712, currentSize=40.25 KB/41220 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 953ms, sequenceid=364, compaction requested=true 2023-04-21 01:53:51,894 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:51,894 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:51,894 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-21 01:53:51,894 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-21 01:53:51,894 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:53:51,894 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:53:51,898 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72919 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-21 01:53:51,898 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 75359 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-21 01:53:51,898 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/B is initiating minor compaction (all files) 2023-04-21 01:53:51,898 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/A is initiating minor compaction (all files) 2023-04-21 01:53:51,898 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/B in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:51,898 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/A in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:51,898 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/cb2aa42de24747de9fd3cc3473ec63ee, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/30e2da97fac6451a9247350c25b12b6a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/de11981e03ea4ebc9d406480c7578199, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/a59cca98e04641ad955f732fbe59657c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/7a749b8b1c274a979edfab794e5df3dd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/b7c969d50f884cf4bf34a136a6d8eb73] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=71.2 K 2023-04-21 01:53:51,898 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b5cd117cbbd84aa6a0ba94ab0c128ba8, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/0807cca5c9134179a1a71f07f50da02e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/44441447fd5146dcbbcb9448d88f959e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b905972c1fbd49a58a426eb0f7c29905, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/4a48a64bba844b598c56189250f3569f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/adb5a5aa3a2f4e89b034fd704b422996] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=73.6 K 2023-04-21 01:53:51,899 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting cb2aa42de24747de9fd3cc3473ec63ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1682042026728 2023-04-21 01:53:51,900 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting b5cd117cbbd84aa6a0ba94ab0c128ba8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1682042026728 2023-04-21 01:53:51,902 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 30e2da97fac6451a9247350c25b12b6a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1682042026850 2023-04-21 01:53:51,902 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 0807cca5c9134179a1a71f07f50da02e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1682042026850 2023-04-21 01:53:51,902 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting de11981e03ea4ebc9d406480c7578199, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1682042028033 2023-04-21 01:53:51,903 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 44441447fd5146dcbbcb9448d88f959e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1682042028033 2023-04-21 01:53:51,903 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting a59cca98e04641ad955f732fbe59657c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682042029323 2023-04-21 01:53:51,904 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting b905972c1fbd49a58a426eb0f7c29905, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682042029323 2023-04-21 01:53:51,904 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 7a749b8b1c274a979edfab794e5df3dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682042029959 2023-04-21 01:53:51,904 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 4a48a64bba844b598c56189250f3569f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682042029959 2023-04-21 01:53:51,904 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting b7c969d50f884cf4bf34a136a6d8eb73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1682042030399 2023-04-21 01:53:51,905 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting adb5a5aa3a2f4e89b034fd704b422996, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1682042030386 2023-04-21 01:53:51,939 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#B#compaction#27 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:51,939 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:51,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:51,946 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:51,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:51,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:51,959 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:51,959 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:51,965 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#A#compaction#28 average throughput is 0.44 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:51,969 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:51,969 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:51,972 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,972 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,974 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,976 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:51,976 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:51,976 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:51,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:51,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:51,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:51,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:51,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:51,988 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:51,988 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:51,990 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:51,992 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:51,992 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:51,992 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:51,993 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:51,994 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-21 01:53:51,995 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:51,995 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:51,995 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:51,995 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:51,995 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:51,995 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:52,006 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/126287b21da74ddd8d9f1e7bc10f438e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/126287b21da74ddd8d9f1e7bc10f438e 2023-04-21 01:53:52,016 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/b801f927d9034a36905920a2d9ad0ab0 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b801f927d9034a36905920a2d9ad0ab0 2023-04-21 01:53:52,026 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/B of 16b5c71a3c47da32fe9f26ec9e41dfb8 into 126287b21da74ddd8d9f1e7bc10f438e(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:52,026 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:52,027 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/B, priority=10, startTime=1682042031894; duration=0sec 2023-04-21 01:53:52,027 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:53:52,027 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-21 01:53:52,028 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/A of 16b5c71a3c47da32fe9f26ec9e41dfb8 into b801f927d9034a36905920a2d9ad0ab0(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:52,028 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:52,028 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/A, priority=10, startTime=1682042031894; duration=0sec 2023-04-21 01:53:52,029 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:52,035 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72919 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-21 01:53:52,035 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/C is initiating minor compaction (all files) 2023-04-21 01:53:52,035 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/C in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:52,036 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/c3fe54704263476cb315207d4e23c1c6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/c500a9a9d8fa40f089d191b93770dcc1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/408896e8c85a49a5a55e1d2fad7e42d9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/eecbfb279e6644898a87caa604a1c345, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/6d4e5f6783174912a7fe06b88da34d9b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/c0d44205feb04bc58e7d3476dac1014c] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=71.2 K 2023-04-21 01:53:52,039 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting c3fe54704263476cb315207d4e23c1c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1682042026728 2023-04-21 01:53:52,041 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting c500a9a9d8fa40f089d191b93770dcc1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1682042026850 2023-04-21 01:53:52,041 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/39d7407cc7c44e7fb1f5ad3907f7bfaa 2023-04-21 01:53:52,042 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 408896e8c85a49a5a55e1d2fad7e42d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1682042028033 2023-04-21 01:53:52,043 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting eecbfb279e6644898a87caa604a1c345, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682042029323 2023-04-21 01:53:52,043 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 6d4e5f6783174912a7fe06b88da34d9b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682042029959 2023-04-21 01:53:52,044 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting c0d44205feb04bc58e7d3476dac1014c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1682042030399 2023-04-21 01:53:52,080 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#C#compaction#31 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:52,104 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/dad73e23304c4dce81ba08091d21fdde 2023-04-21 01:53:52,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:52,116 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:52,171 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/57be0fc4b4d44ec68ee2b11c44790d72 2023-04-21 01:53:52,187 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/39d7407cc7c44e7fb1f5ad3907f7bfaa as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/39d7407cc7c44e7fb1f5ad3907f7bfaa 2023-04-21 01:53:52,203 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/39d7407cc7c44e7fb1f5ad3907f7bfaa, entries=150, sequenceid=373, filesize=11.9 K 2023-04-21 01:53:52,205 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/dad73e23304c4dce81ba08091d21fdde as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/dad73e23304c4dce81ba08091d21fdde 2023-04-21 01:53:52,227 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/dad73e23304c4dce81ba08091d21fdde, entries=150, sequenceid=373, filesize=11.9 K 2023-04-21 01:53:52,228 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/57be0fc4b4d44ec68ee2b11c44790d72 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/57be0fc4b4d44ec68ee2b11c44790d72 2023-04-21 01:53:52,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042032174, max timestamp=1682042032227Num uniques 859; 2023-04-21 01:53:52,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:53:52,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042032174, max timestamp=1682042032227Num uniques 859; 2023-04-21 01:53:52,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:53:52,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042032174, max timestamp=1682042032227Num uniques 859; 2023-04-21 01:53:52,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:53:52,244 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/57be0fc4b4d44ec68ee2b11c44790d72, entries=150, sequenceid=373, filesize=11.9 K 2023-04-21 01:53:52,245 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=40.25 KB/41220 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 252ms, sequenceid=373, compaction requested=false 2023-04-21 01:53:52,245 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:52,245 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:52,245 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:52,245 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:52,245 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,245 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:52,245 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:52,247 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,247 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:52,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:52,247 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,248 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:52,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:52,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:52,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:52,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:52,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:52,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@521f574c[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:52,250 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:52,250 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,252 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,252 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:52,252 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:52,252 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:52,252 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,252 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:52,255 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,255 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:52,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:52,255 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:52,256 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:52,256 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=53.67 KB heapSize=145.92 KB 2023-04-21 01:53:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:52,256 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:52,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:52,256 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:52,258 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:52,258 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:52,258 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:52,259 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:52,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:52,263 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:52,263 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:52,263 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:52,263 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:52,263 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:52,266 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,266 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:52,266 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:52,267 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:52,269 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,278 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,278 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:52,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:52,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:52,279 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:52,279 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:52,279 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:52,279 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,279 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,279 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,279 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,279 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:52,279 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:52,279 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:52,279 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:52,279 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:52,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:52,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:52,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042032328, max timestamp=1682042032342Num uniques 859; 2023-04-21 01:53:52,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042032328, max timestamp=1682042032342Num uniques 859; 2023-04-21 01:53:52,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:53:52,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042032328, max timestamp=1682042032342Num uniques 859; 2023-04-21 01:53:52,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:53:52,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:53:52,379 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:52,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:52,383 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:52,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:52,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:52,384 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:52,385 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:52,386 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:52,386 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:52,386 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,386 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:52,393 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,393 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:52,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:52,393 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,393 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:52,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:52,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:52,394 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:52,395 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:52,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:52,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:52,397 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:52,398 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:52,398 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:52,398 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:52,398 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:52,398 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:52,398 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:52,398 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:52,399 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:52,399 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,399 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:52,399 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:52,405 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:52,405 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:52,406 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,406 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:52,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:52,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:52,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d973a32[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:52,407 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:52,407 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,414 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,416 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:52,416 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:52,416 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:52,418 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:52,418 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,418 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:52,418 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:52,420 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:52,420 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:52,420 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:52,420 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:52,420 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:52,420 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:52,420 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:52,422 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,422 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:52,422 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:52,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:52,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:52,423 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:52,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:52,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:52,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,430 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,434 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:52,434 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:52,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:52,434 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:52,435 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:52,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:52,435 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:52,434 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:52,434 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:52,440 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:52,440 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,440 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,440 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,440 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,434 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:52,435 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:52,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:52,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:52,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:52,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59276 deadline: 1682042092456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,541 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:52,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:52,546 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/2a32e09968d84d61b2c597c3e65f7f03 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/2a32e09968d84d61b2c597c3e65f7f03 2023-04-21 01:53:52,546 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:52,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:52,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042092547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:52,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:52,552 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:52,552 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:52,552 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:52,552 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:52,552 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,553 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:52,554 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:52,554 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:52,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:52,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:52,555 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,555 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:52,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:52,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042092552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:52,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:52,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:52,556 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/C of 16b5c71a3c47da32fe9f26ec9e41dfb8 into 2a32e09968d84d61b2c597c3e65f7f03(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:52,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:52,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:52,557 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:52,557 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:52,557 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:52,556 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:52,557 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:52,557 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:52,557 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/C, priority=10, startTime=1682042031894; duration=0sec 2023-04-21 01:53:52,557 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:52,558 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:52,558 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:52,558 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:52,558 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:52,558 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:52,558 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:52,558 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,558 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:52,558 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:52,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:52,560 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042092555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:52,560 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,561 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,561 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:52,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:52,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:52,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:52,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aacad07[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:52,563 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:52,563 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:52,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042092557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:52,565 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,565 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:52,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59276 deadline: 1682042092564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,565 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,565 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:52,565 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:52,565 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:52,568 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,568 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:52,568 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:52,568 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:52,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:52,571 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:52,571 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:52,571 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:52,571 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:52,571 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:52,572 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,572 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,572 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:52,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:52,573 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:52,577 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:52,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:52,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:52,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:52,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:52,582 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:52,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:52,582 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:52,582 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:52,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:52,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,582 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:52,582 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:52,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:52,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:52,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:52,683 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:52,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:52,686 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:52,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:52,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:52,689 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:52,689 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:52,689 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:52,689 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:52,689 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,689 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:52,692 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:52,692 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:52,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:52,693 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,693 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:52,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:52,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:52,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:52,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:52,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:52,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:52,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:52,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:52,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:52,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:52,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:52,694 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:52,695 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:52,695 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:52,695 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:52,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:52,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:52,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:52,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:52,696 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:52,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:52,697 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,697 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:52,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:52,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:52,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:52,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6177d06c[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:52,698 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:52,699 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,700 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,700 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:52,700 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:52,700 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:52,700 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,700 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:52,702 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,702 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:52,702 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:52,702 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:52,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:52,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:52,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:52,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:52,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:52,705 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:52,705 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:52,705 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:52,705 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:52,705 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:52,707 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,708 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,708 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:52,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:52,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:52,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:52,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:52,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:52,709 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:52,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:52,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:52,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:52,710 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:52,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,714 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/f4dc85d78cf34cb881f1dc540f1b0d33 2023-04-21 01:53:52,714 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,714 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:52,714 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:52,714 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,714 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,714 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:52,714 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,714 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:52,714 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:52,714 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:52,714 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:52,714 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:52,715 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:52,715 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:52,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:52,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:52,743 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/b3acadcd4aa34e4f9e821137527a401f 2023-04-21 01:53:52,761 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/a04c2bacacae48e5bf4a47ed17c88fea 2023-04-21 01:53:52,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:52,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59276 deadline: 1682042092766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,769 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/f4dc85d78cf34cb881f1dc540f1b0d33 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/f4dc85d78cf34cb881f1dc540f1b0d33 2023-04-21 01:53:52,776 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/f4dc85d78cf34cb881f1dc540f1b0d33, entries=200, sequenceid=400, filesize=14.3 K 2023-04-21 01:53:52,778 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/b3acadcd4aa34e4f9e821137527a401f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/b3acadcd4aa34e4f9e821137527a401f 2023-04-21 01:53:52,787 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/b3acadcd4aa34e4f9e821137527a401f, entries=150, sequenceid=400, filesize=11.9 K 2023-04-21 01:53:52,788 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/a04c2bacacae48e5bf4a47ed17c88fea as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/a04c2bacacae48e5bf4a47ed17c88fea 2023-04-21 01:53:52,800 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/a04c2bacacae48e5bf4a47ed17c88fea, entries=150, sequenceid=400, filesize=11.9 K 2023-04-21 01:53:52,801 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.88 KB/149376, currentSize=134.18 KB/137400 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 545ms, sequenceid=400, compaction requested=true 2023-04-21 01:53:52,801 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:52,801 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:52,801 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:53:52,801 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:53:52,801 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:53:52,801 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:53:52,802 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39306 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:53:52,803 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/A is initiating minor compaction (all files) 2023-04-21 01:53:52,803 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36866 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:53:52,803 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/A in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:52,803 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/B is initiating minor compaction (all files) 2023-04-21 01:53:52,803 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/B in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:52,803 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b801f927d9034a36905920a2d9ad0ab0, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/39d7407cc7c44e7fb1f5ad3907f7bfaa, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/f4dc85d78cf34cb881f1dc540f1b0d33] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=38.4 K 2023-04-21 01:53:52,803 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/126287b21da74ddd8d9f1e7bc10f438e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/dad73e23304c4dce81ba08091d21fdde, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/b3acadcd4aa34e4f9e821137527a401f] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=36.0 K 2023-04-21 01:53:52,803 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting b801f927d9034a36905920a2d9ad0ab0, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1682042030399 2023-04-21 01:53:52,804 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 126287b21da74ddd8d9f1e7bc10f438e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1682042030399 2023-04-21 01:53:52,806 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 39d7407cc7c44e7fb1f5ad3907f7bfaa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1682042030953 2023-04-21 01:53:52,806 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting dad73e23304c4dce81ba08091d21fdde, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1682042030953 2023-04-21 01:53:52,807 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting f4dc85d78cf34cb881f1dc540f1b0d33, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1682042032202 2023-04-21 01:53:52,807 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting b3acadcd4aa34e4f9e821137527a401f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1682042032227 2023-04-21 01:53:52,815 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:52,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:52,820 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:52,833 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#A#compaction#36 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:52,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:52,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:52,840 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:52,840 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:52,840 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:52,840 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:52,841 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,841 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:52,842 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:52,842 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:52,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:52,843 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:52,843 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:52,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:52,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:52,844 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#B#compaction#37 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:52,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:52,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:52,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:52,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:52,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:52,851 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:52,851 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:52,851 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:52,851 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:52,851 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:52,851 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:52,851 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:52,851 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-21 01:53:52,852 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:52,852 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:52,852 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:52,852 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:52,852 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:52,852 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:52,860 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/c6970e5e10bb43ab9e249f51377cabef as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/c6970e5e10bb43ab9e249f51377cabef 2023-04-21 01:53:52,874 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/A of 16b5c71a3c47da32fe9f26ec9e41dfb8 into c6970e5e10bb43ab9e249f51377cabef(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:52,874 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:52,874 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/A, priority=13, startTime=1682042032801; duration=0sec 2023-04-21 01:53:52,875 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/2fd310eb7c6b43a2994cf0c995750338 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/2fd310eb7c6b43a2994cf0c995750338 2023-04-21 01:53:52,875 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:53:52,875 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:53:52,876 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36866 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:53:52,876 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/C is initiating minor compaction (all files) 2023-04-21 01:53:52,877 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/C in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:52,877 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/2a32e09968d84d61b2c597c3e65f7f03, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/57be0fc4b4d44ec68ee2b11c44790d72, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/a04c2bacacae48e5bf4a47ed17c88fea] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=36.0 K 2023-04-21 01:53:52,877 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 2a32e09968d84d61b2c597c3e65f7f03, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1682042030399 2023-04-21 01:53:52,879 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 57be0fc4b4d44ec68ee2b11c44790d72, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1682042030953 2023-04-21 01:53:52,880 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting a04c2bacacae48e5bf4a47ed17c88fea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1682042032227 2023-04-21 01:53:52,906 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#C#compaction#39 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:52,907 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/B of 16b5c71a3c47da32fe9f26ec9e41dfb8 into 2fd310eb7c6b43a2994cf0c995750338(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:52,908 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:52,908 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/B, priority=13, startTime=1682042032801; duration=0sec 2023-04-21 01:53:52,908 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:52,910 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/94b78c7f9692459d97a3badd102551db 2023-04-21 01:53:52,932 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/ff700b092a124cbfbbd6b55cd62960ec as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/ff700b092a124cbfbbd6b55cd62960ec 2023-04-21 01:53:52,940 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/C of 16b5c71a3c47da32fe9f26ec9e41dfb8 into ff700b092a124cbfbbd6b55cd62960ec(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:52,940 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:52,940 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/C, priority=13, startTime=1682042032801; duration=0sec 2023-04-21 01:53:52,940 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:52,949 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/cb8e7b80d26849618cdac024eb31921b 2023-04-21 01:53:52,971 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/6a243df1b5dd4e06b55f7ad64e7b2768 2023-04-21 01:53:52,989 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/94b78c7f9692459d97a3badd102551db as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/94b78c7f9692459d97a3badd102551db 2023-04-21 01:53:52,997 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/94b78c7f9692459d97a3badd102551db, entries=150, sequenceid=438, filesize=11.9 K 2023-04-21 01:53:52,999 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/cb8e7b80d26849618cdac024eb31921b as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/cb8e7b80d26849618cdac024eb31921b 2023-04-21 01:53:53,022 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/cb8e7b80d26849618cdac024eb31921b, entries=150, sequenceid=438, filesize=11.9 K 2023-04-21 01:53:53,024 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/6a243df1b5dd4e06b55f7ad64e7b2768 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/6a243df1b5dd4e06b55f7ad64e7b2768 2023-04-21 01:53:53,035 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/6a243df1b5dd4e06b55f7ad64e7b2768, entries=150, sequenceid=438, filesize=11.9 K 2023-04-21 01:53:53,037 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 186ms, sequenceid=438, compaction requested=false 2023-04-21 01:53:53,037 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:53,037 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:53,037 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:53,041 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,041 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:53,042 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,042 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:53,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:53,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@497a5e39[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:53,045 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:53,045 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,047 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,047 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:53,047 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:53,047 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:53,048 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,048 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:53,049 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:53,049 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:53,049 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:53,049 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:53,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:53,053 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:53,053 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:53,053 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:53,053 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:53,053 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:53,055 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,055 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:53,055 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:53,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:53,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:53,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,057 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,059 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:53,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,062 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:53,062 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:53,062 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,062 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,062 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:53,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:53,062 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:53,063 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:53,063 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:53,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:53,062 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,063 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,063 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,063 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,063 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,062 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:53,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:53,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:53,093 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-21 01:53:53,093 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:53,094 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:53,094 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:53,094 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:53,094 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:53,094 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:53,105 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/cc504ad8b2df41a9bfee6a0c011c8b10 2023-04-21 01:53:53,133 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/3de31e6e4a464b0aa053849e29f03206 2023-04-21 01:53:53,160 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/fad0173f016544eca87ee7d593ddeb8f 2023-04-21 01:53:53,163 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:53,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:53,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682042033124, max timestamp=1682042033163Num uniques 859; 2023-04-21 01:53:53,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-21 01:53:53,167 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:53,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682042033124, max timestamp=1682042033163Num uniques 859; 2023-04-21 01:53:53,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-21 01:53:53,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682042033124, max timestamp=1682042033163Num uniques 859; 2023-04-21 01:53:53,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-21 01:53:53,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:53,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:53,169 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:53,169 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:53,169 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:53,169 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:53,169 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,169 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,171 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:53,172 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:53,172 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,172 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:53,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:53,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:53,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:53,174 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:53,175 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:53,175 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:53,178 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/cc504ad8b2df41a9bfee6a0c011c8b10 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/cc504ad8b2df41a9bfee6a0c011c8b10 2023-04-21 01:53:53,180 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:53,180 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:53,180 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:53,180 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:53,181 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:53,181 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:53,181 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:53,181 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:53,181 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,181 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:53,181 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:53,183 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,183 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:53,184 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,184 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,187 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/cc504ad8b2df41a9bfee6a0c011c8b10, entries=150, sequenceid=451, filesize=11.9 K 2023-04-21 01:53:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:53,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:53,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4db8bb8e[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:53,188 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:53,188 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,189 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:53,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:53,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:53,190 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,190 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:53,190 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/3de31e6e4a464b0aa053849e29f03206 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/3de31e6e4a464b0aa053849e29f03206 2023-04-21 01:53:53,191 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:53,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:53,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:53,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:53,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:53,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:53,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:53,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:53,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:53,197 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:53,198 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:53,198 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:53,198 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:53,198 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:53,201 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,201 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:53,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,201 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:53,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:53,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:53,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:53,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:53,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,202 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,204 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,207 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/3de31e6e4a464b0aa053849e29f03206, entries=150, sequenceid=451, filesize=11.9 K 2023-04-21 01:53:53,208 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/fad0173f016544eca87ee7d593ddeb8f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/fad0173f016544eca87ee7d593ddeb8f 2023-04-21 01:53:53,214 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:53,214 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:53,214 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:53,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:53,214 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,214 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,214 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,214 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,215 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,215 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,214 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:53,215 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:53,214 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:53,215 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:53,215 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:53,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:53,232 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/fad0173f016544eca87ee7d593ddeb8f, entries=150, sequenceid=451, filesize=11.9 K 2023-04-21 01:53:53,236 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=120.76 KB/123660 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 142ms, sequenceid=451, compaction requested=true 2023-04-21 01:53:53,236 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:53,236 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:53,236 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:53:53,236 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:53,236 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:53:53,236 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:53:53,238 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:53:53,239 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:53:53,239 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/B is initiating minor compaction (all files) 2023-04-21 01:53:53,239 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/A is initiating minor compaction (all files) 2023-04-21 01:53:53,239 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/B in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:53,239 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/A in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:53,239 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/2fd310eb7c6b43a2994cf0c995750338, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/cb8e7b80d26849618cdac024eb31921b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/3de31e6e4a464b0aa053849e29f03206] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=36.1 K 2023-04-21 01:53:53,239 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/c6970e5e10bb43ab9e249f51377cabef, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/94b78c7f9692459d97a3badd102551db, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/cc504ad8b2df41a9bfee6a0c011c8b10] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=36.1 K 2023-04-21 01:53:53,240 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 2fd310eb7c6b43a2994cf0c995750338, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1682042032227 2023-04-21 01:53:53,241 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting c6970e5e10bb43ab9e249f51377cabef, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1682042032227 2023-04-21 01:53:53,242 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting cb8e7b80d26849618cdac024eb31921b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682042032424 2023-04-21 01:53:53,242 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 94b78c7f9692459d97a3badd102551db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682042032424 2023-04-21 01:53:53,242 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=127.47 KB heapSize=351 KB 2023-04-21 01:53:53,242 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 3de31e6e4a464b0aa053849e29f03206, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1682042033082 2023-04-21 01:53:53,242 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:53,243 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:53,243 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:53,243 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:53,243 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:53,243 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:53,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:53,243 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting cc504ad8b2df41a9bfee6a0c011c8b10, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1682042033082 2023-04-21 01:53:53,262 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#B#compaction#46 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:53,269 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#A#compaction#47 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:53,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:53,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59276 deadline: 1682042093294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,315 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:53,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:53,318 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:53,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:53,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:53,320 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:53,320 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:53,320 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:53,320 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:53,320 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,321 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,322 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:53,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:53,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:53,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:53,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:53,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:53,324 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:53,324 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:53,324 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:53,324 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:53,324 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:53,324 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:53,324 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:53,325 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:53,325 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:53,325 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:53,325 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:53,325 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,325 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:53,325 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:53,327 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,327 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:53,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:53,327 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:53,328 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:53,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:53,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:53,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:53,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3203f3ab[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:53,331 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:53,332 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,333 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,333 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:53,333 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:53,333 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:53,333 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,333 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:53,334 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:53,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:53,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:53,335 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:53,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:53,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:53,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:53,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:53,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:53,338 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:53,338 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:53,338 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:53,338 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:53,338 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:53,340 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:53,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:53,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:53,341 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,342 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,342 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:53,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,345 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,349 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,349 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,349 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,349 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,349 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,349 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,349 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:53,352 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:53,352 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:53,352 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:53,352 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:53,352 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:53,359 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:53,359 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:53,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:53,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:53,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:53,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59276 deadline: 1682042093400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,453 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:53,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:53,456 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:53,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:53,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:53,458 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:53,458 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:53,458 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:53,458 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:53,458 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,458 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:53,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:53,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,461 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,461 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,461 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:53,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:53,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:53,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:53,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:53,462 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:53,462 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:53,462 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:53,462 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:53,462 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:53,462 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:53,462 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:53,462 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:53,462 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:53,462 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:53,462 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:53,462 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,463 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:53,463 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:53,464 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,464 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:53,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:53,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6da8a137[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:53,466 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:53,466 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,468 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,468 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:53,468 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,468 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:53,468 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:53,468 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:53,470 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,470 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:53,470 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:53,471 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:53,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:53,473 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:53,473 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:53,473 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:53,473 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:53,473 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:53,476 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,476 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:53,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,477 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:53,478 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,480 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,482 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:53,482 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:53,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:53,483 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:53,482 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,483 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:53,483 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:53,483 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,483 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,483 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:53,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:53,483 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:53,483 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:53,483 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,483 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,483 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,583 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:53,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:53,587 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:53,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:53,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:53,589 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:53,589 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:53,589 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:53,589 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:53,589 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,589 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,591 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:53,591 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:53,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:53,591 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,592 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:53,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:53,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:53,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:53,593 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:53,593 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:53,594 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:53,594 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:53,594 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:53,594 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:53,594 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:53,594 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:53,594 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:53,594 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:53,594 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:53,594 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,594 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:53,594 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:53,597 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,597 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:53,598 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,598 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:53,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:53,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:53,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76b8a12d[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:53,600 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:53,600 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,602 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,602 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:53,602 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:53,602 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:53,603 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,603 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:53,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:53,604 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:53,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59276 deadline: 1682042093602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,604 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,604 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:53,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:53,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:53,604 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:53,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:53,610 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:53,610 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:53,611 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:53,611 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:53,611 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:53,612 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,612 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:53,612 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:53,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:53,613 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,614 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,618 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:53,618 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:53,618 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,618 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:53,618 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,618 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:53,618 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:53,618 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:53,618 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:53,618 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,619 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:53,619 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:53,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:53,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:53,679 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/7536aa07d5ed4c5c8c9f666b72289e66 2023-04-21 01:53:53,695 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/51f172d9138544a3a80f2f173a464790 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/51f172d9138544a3a80f2f173a464790 2023-04-21 01:53:53,699 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/244c34418771484b926c51928969a40a 2023-04-21 01:53:53,702 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/ac38b0648e1c4dd29bd7e4594c2f5f35 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/ac38b0648e1c4dd29bd7e4594c2f5f35 2023-04-21 01:53:53,706 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/B of 16b5c71a3c47da32fe9f26ec9e41dfb8 into 51f172d9138544a3a80f2f173a464790(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:53,706 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:53,706 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/B, priority=13, startTime=1682042033236; duration=0sec 2023-04-21 01:53:53,706 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:53:53,707 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:53:53,709 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:53:53,709 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/C is initiating minor compaction (all files) 2023-04-21 01:53:53,709 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/C in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:53,709 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/ff700b092a124cbfbbd6b55cd62960ec, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/6a243df1b5dd4e06b55f7ad64e7b2768, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/fad0173f016544eca87ee7d593ddeb8f] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=36.1 K 2023-04-21 01:53:53,710 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/A of 16b5c71a3c47da32fe9f26ec9e41dfb8 into ac38b0648e1c4dd29bd7e4594c2f5f35(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:53,710 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:53,710 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/A, priority=13, startTime=1682042033236; duration=0sec 2023-04-21 01:53:53,710 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:53,711 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting ff700b092a124cbfbbd6b55cd62960ec, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1682042032227 2023-04-21 01:53:53,713 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 6a243df1b5dd4e06b55f7ad64e7b2768, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1682042032424 2023-04-21 01:53:53,713 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting fad0173f016544eca87ee7d593ddeb8f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1682042033082 2023-04-21 01:53:53,719 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:53,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:53,722 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:53,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:53,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:53,732 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:53,732 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:53,732 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:53,732 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:53,733 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,733 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,734 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#C#compaction#50 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:53,734 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,735 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:53,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:53,736 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,736 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:53,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:53,738 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:53,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:53,739 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:53,739 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:53,740 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:53,740 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:53,740 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:53,740 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:53,740 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:53,740 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,740 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:53,740 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:53,744 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,744 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:53,745 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:53,745 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ed400bf[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:53,747 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:53,747 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,748 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/9404233302cd4638998c17b3a5260dad as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/9404233302cd4638998c17b3a5260dad 2023-04-21 01:53:53,750 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,750 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:53,750 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:53,750 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:53,750 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,750 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:53,752 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:53,752 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:53,752 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:53,752 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:53,755 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/C of 16b5c71a3c47da32fe9f26ec9e41dfb8 into 9404233302cd4638998c17b3a5260dad(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:53,755 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:53,755 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/C, priority=13, startTime=1682042033236; duration=0sec 2023-04-21 01:53:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,755 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:53,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:53,756 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:53,756 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:53,756 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:53,756 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:53,756 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:53,758 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,758 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,759 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:53,759 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,761 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:53,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,763 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,764 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,764 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,764 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,764 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,764 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:53,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:53,764 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:53,764 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:53,764 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,764 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:53,764 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:53,764 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:53,764 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:53,764 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:53,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:53,864 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:53,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:53,869 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:53,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:53,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:53,870 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:53,870 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:53,870 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:53,870 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:53,871 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,871 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,873 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:53,873 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:53,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:53,874 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,874 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:53,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:53,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:53,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:53,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:53,875 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:53,875 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:53,875 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:53,875 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:53,875 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:53,875 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:53,875 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:53,876 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:53,876 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:53,876 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:53,876 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:53,876 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,876 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:53,876 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:53,877 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,877 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:53,878 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,878 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:53,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:53,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:53,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53e73a65[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:53,880 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:53,880 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,881 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,881 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:53,882 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:53,882 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:53,882 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,882 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:53,883 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,883 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:53,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,883 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:53,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:53,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:53,884 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:53,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:53,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:53,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:53,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:53,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:53,887 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:53,887 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:53,887 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:53,887 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:53,887 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:53,890 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,890 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:53,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:53,891 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:53,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:53,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:53,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:53,892 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:53,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,894 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:53,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:53,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,898 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:53,898 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:53,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:53,898 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:53,898 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:53,898 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,898 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:53,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:53,898 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:53,898 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:53,899 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:53,898 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:53,899 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,899 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,899 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:53,899 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:53,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:53,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59276 deadline: 1682042093906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:53,999 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:53,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:54,002 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:54,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:54,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:54,004 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:54,004 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:54,004 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:54,004 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:54,004 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,004 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,006 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:54,006 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:54,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:54,006 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,006 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:54,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:54,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:54,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:54,007 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:54,008 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:54,008 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:54,008 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:54,008 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:54,008 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:54,008 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:54,008 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:54,008 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:54,008 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:54,008 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:54,008 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,008 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:54,008 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:54,010 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,010 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:54,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:54,010 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,010 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:54,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:54,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bee9837[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:54,013 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:54,014 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,017 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,017 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:54,017 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:54,017 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:54,018 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,018 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:54,019 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,019 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:54,019 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:54,019 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:54,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:54,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:54,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:54,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:54,022 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:54,022 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:54,022 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:54,022 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:54,022 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:54,024 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,024 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:54,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,024 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:54,024 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,026 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,029 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:54,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:54,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,029 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:54,029 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:54,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:54,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:54,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:54,029 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:54,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,029 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:54,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:54,129 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:54,130 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/f26105d4af68439ba4bcebacb3a886ce 2023-04-21 01:53:54,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:54,136 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:54,138 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/7536aa07d5ed4c5c8c9f666b72289e66 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/7536aa07d5ed4c5c8c9f666b72289e66 2023-04-21 01:53:54,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:54,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:54,139 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:54,139 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:54,139 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:54,139 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:54,140 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,140 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,143 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:54,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:54,143 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:54,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,144 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,144 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:54,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:54,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:54,145 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:54,145 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:54,146 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:54,146 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:54,146 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:54,146 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:54,146 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:54,146 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:54,146 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:54,146 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:54,147 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:54,147 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:54,150 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/7536aa07d5ed4c5c8c9f666b72289e66, entries=250, sequenceid=484, filesize=16.7 K 2023-04-21 01:53:54,152 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/244c34418771484b926c51928969a40a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/244c34418771484b926c51928969a40a 2023-04-21 01:53:54,153 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:54,154 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:54,154 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,154 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:54,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:54,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48ecefae[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:54,157 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:54,158 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,159 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/244c34418771484b926c51928969a40a, entries=150, sequenceid=484, filesize=11.9 K 2023-04-21 01:53:54,161 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/f26105d4af68439ba4bcebacb3a886ce as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/f26105d4af68439ba4bcebacb3a886ce 2023-04-21 01:53:54,161 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,162 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:54,162 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:54,162 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:54,162 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,162 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:54,165 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,165 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:54,165 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:54,165 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:54,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,175 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/f26105d4af68439ba4bcebacb3a886ce, entries=150, sequenceid=484, filesize=11.9 K 2023-04-21 01:53:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:54,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:54,176 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:54,176 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:54,176 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:54,176 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:54,176 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:54,176 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~350.95 KB/359376, currentSize=60.38 KB/61830 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 934ms, sequenceid=484, compaction requested=false 2023-04-21 01:53:54,176 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:54,178 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,178 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:54,178 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:54,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:54,179 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:54,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,182 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:54,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,188 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:54,188 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:54,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:54,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:54,190 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,190 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,190 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,190 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,190 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,190 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,190 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:54,193 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:54,193 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:54,194 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:54,194 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:54,294 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:54,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:54,320 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:54,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:54,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:54,323 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:54,323 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:54,323 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:54,323 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:54,324 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,324 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,327 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:54,327 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:54,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:54,328 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,328 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:54,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:54,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:54,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:54,329 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:54,329 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:54,329 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:54,329 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:54,330 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:54,330 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:54,331 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:54,331 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-21 01:53:54,331 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:54,331 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:54,331 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:54,331 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:54,331 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:54,332 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:54,352 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/9f3dcd3239cd44e9bcc883abddbd8a0d 2023-04-21 01:53:54,377 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/ae8007396436475eab082aef1d03a7ea 2023-04-21 01:53:54,417 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:54,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:54,435 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/78542f67720249dcb31e3c798ded7510 2023-04-21 01:53:54,446 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/9f3dcd3239cd44e9bcc883abddbd8a0d as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/9f3dcd3239cd44e9bcc883abddbd8a0d 2023-04-21 01:53:54,457 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/9f3dcd3239cd44e9bcc883abddbd8a0d, entries=150, sequenceid=499, filesize=11.9 K 2023-04-21 01:53:54,461 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/ae8007396436475eab082aef1d03a7ea as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/ae8007396436475eab082aef1d03a7ea 2023-04-21 01:53:54,468 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/ae8007396436475eab082aef1d03a7ea, entries=150, sequenceid=499, filesize=11.9 K 2023-04-21 01:53:54,470 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/78542f67720249dcb31e3c798ded7510 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/78542f67720249dcb31e3c798ded7510 2023-04-21 01:53:54,478 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/78542f67720249dcb31e3c798ded7510, entries=150, sequenceid=499, filesize=11.9 K 2023-04-21 01:53:54,481 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=67.09 KB/68700 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 149ms, sequenceid=499, compaction requested=true 2023-04-21 01:53:54,481 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:54,481 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:54,481 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:54,495 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:54,495 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,495 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:54,495 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:54,499 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:54,500 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,500 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,500 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:54,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:54,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:54,514 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-21 01:53:54,514 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:54,515 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:54,515 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:54,515 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:54,515 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:54,515 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:54,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:54,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16e3dff4[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:54,518 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:54,518 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,521 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,521 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:54,521 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:54,521 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:54,523 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:54,523 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:54,523 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:54,523 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,525 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:54,525 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:54,525 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:54,525 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:54,525 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:54,525 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:54,525 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:54,527 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,527 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:54,527 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:54,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:54,531 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,533 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,537 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:54,537 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:54,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:54,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:54,538 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,538 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,538 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,538 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:54,538 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,538 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:54,538 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:54,538 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,539 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,539 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,539 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:54,539 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:54,576 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/320f4a48849d475d99606bc7541b6ce6 2023-04-21 01:53:54,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:54,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59276 deadline: 1682042094570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:54,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:54,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042094573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042094574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:54,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042094576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:54,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042094582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,594 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4193 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:53:54,621 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/834793b7ccd4430596eea3cbf5ccfc57 2023-04-21 01:53:54,639 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:54,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:54,643 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:54,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:54,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:54,652 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:54,652 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:54,652 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:54,652 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:54,660 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,661 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,662 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:54,662 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:54,663 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,663 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:54,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:54,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:54,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:54,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:54,676 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:54,676 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:54,676 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:54,676 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:54,676 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:54,676 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:54,677 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:54,677 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:54,677 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:54,677 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:54,677 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:54,677 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,677 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:54,677 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:54,679 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,680 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,680 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:54,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:54,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:54,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59276 deadline: 1682042094681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042094682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,685 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:54,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042094684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,688 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:54,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:54,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c3dfb5f[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:54,691 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:54,691 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,694 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:54,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:54,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:54,695 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,695 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:54,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:54,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042094694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,698 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,698 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:54,698 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:54,698 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:54,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:54,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:54,706 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:54,706 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:54,707 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:54,707 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:54,707 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:54,711 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,711 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:54,711 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:54,712 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:54,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:54,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,714 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,716 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,716 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,716 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:54,716 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:54,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:54,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:54,716 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:54,716 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:54,717 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:54,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:54,717 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:54,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:54,716 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,717 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:54,717 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,717 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,718 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,718 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,817 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:54,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:54,820 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:54,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:54,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:54,822 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:54,822 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:54,822 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:54,822 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:54,823 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,823 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,825 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,825 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:54,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:54,825 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,825 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:54,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:54,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:54,827 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:54,827 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:54,827 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:54,827 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:54,827 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:54,827 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:54,827 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:54,827 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:54,827 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:54,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,835 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:54,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:54,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@731debcf[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:54,837 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:54,838 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,839 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:54,839 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:54,839 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,839 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:54,839 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:54,845 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:54,845 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,845 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:54,845 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:54,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:54,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:54,850 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:54,850 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:54,850 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:54,850 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:54,850 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:54,855 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,855 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:54,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,855 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:54,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:54,856 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,859 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,862 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:54,862 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:54,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:54,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:54,862 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,862 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,862 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,862 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,862 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,862 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:54,862 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,863 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:54,863 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:54,863 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:54,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:54,863 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:54,863 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:54,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:54,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59276 deadline: 1682042094887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:54,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:54,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59272 deadline: 1682042094888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042094889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:54,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59312 deadline: 1682042094899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,963 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:54,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:54,968 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:54,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:54,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:54,974 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:54,974 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:54,974 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:54,974 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:54,974 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,974 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,978 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,978 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:54,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:54,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:54,978 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,978 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:54,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:54,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:54,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:54,985 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:54,985 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:54,985 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:54,992 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:54,992 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:54,992 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:54,992 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:54,992 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:54,992 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:54,992 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:54,992 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:54,992 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,992 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:54,993 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:54,995 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:54,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:54,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c23fb9d[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:54,998 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:54,998 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,001 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,001 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:55,001 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:55,001 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:55,004 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,004 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:55,005 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:55,005 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:55,005 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:55,008 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:55,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:55,015 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:55,015 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:55,015 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:55,015 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:55,016 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:55,018 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,018 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:55,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:55,018 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:55,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:55,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:55,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:55,019 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:55,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,022 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:55,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,025 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:55,025 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,025 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:55,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:55,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:55,025 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:55,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:55,025 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:55,025 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,025 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:55,025 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:55,025 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,025 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:55,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:55,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:55,026 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,026 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:55,026 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:55,026 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,039 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/8e387eee7fd8445cb664c04937f1d3a7 2023-04-21 01:53:55,047 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/320f4a48849d475d99606bc7541b6ce6 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/320f4a48849d475d99606bc7541b6ce6 2023-04-21 01:53:55,055 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/320f4a48849d475d99606bc7541b6ce6, entries=150, sequenceid=514, filesize=11.9 K 2023-04-21 01:53:55,057 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/834793b7ccd4430596eea3cbf5ccfc57 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/834793b7ccd4430596eea3cbf5ccfc57 2023-04-21 01:53:55,066 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/834793b7ccd4430596eea3cbf5ccfc57, entries=150, sequenceid=514, filesize=11.9 K 2023-04-21 01:53:55,067 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/8e387eee7fd8445cb664c04937f1d3a7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/8e387eee7fd8445cb664c04937f1d3a7 2023-04-21 01:53:55,074 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/8e387eee7fd8445cb664c04937f1d3a7, entries=150, sequenceid=514, filesize=11.9 K 2023-04-21 01:53:55,075 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 561ms, sequenceid=514, compaction requested=true 2023-04-21 01:53:55,075 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:55,076 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:55,076 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:53:55,076 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:53:55,076 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:53:55,076 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:53:55,078 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54136 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:53:55,078 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/A is initiating minor compaction (all files) 2023-04-21 01:53:55,078 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/A in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:55,078 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/ac38b0648e1c4dd29bd7e4594c2f5f35, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/7536aa07d5ed4c5c8c9f666b72289e66, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/9f3dcd3239cd44e9bcc883abddbd8a0d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/320f4a48849d475d99606bc7541b6ce6] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=52.9 K 2023-04-21 01:53:55,078 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:53:55,078 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/B is initiating minor compaction (all files) 2023-04-21 01:53:55,078 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/B in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:55,079 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/51f172d9138544a3a80f2f173a464790, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/244c34418771484b926c51928969a40a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/ae8007396436475eab082aef1d03a7ea, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/834793b7ccd4430596eea3cbf5ccfc57] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=48.1 K 2023-04-21 01:53:55,080 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 51f172d9138544a3a80f2f173a464790, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1682042033082 2023-04-21 01:53:55,081 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting ac38b0648e1c4dd29bd7e4594c2f5f35, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1682042033082 2023-04-21 01:53:55,081 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 244c34418771484b926c51928969a40a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682042033218 2023-04-21 01:53:55,081 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 7536aa07d5ed4c5c8c9f666b72289e66, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682042033189 2023-04-21 01:53:55,082 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting ae8007396436475eab082aef1d03a7ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1682042033268 2023-04-21 01:53:55,082 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 9f3dcd3239cd44e9bcc883abddbd8a0d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1682042033268 2023-04-21 01:53:55,082 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 834793b7ccd4430596eea3cbf5ccfc57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1682042034450 2023-04-21 01:53:55,083 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 320f4a48849d475d99606bc7541b6ce6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1682042034450 2023-04-21 01:53:55,101 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#B#compaction#57 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:55,108 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#A#compaction#58 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:55,126 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:55,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:55,130 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:55,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:55,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:55,133 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:55,133 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:55,133 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:55,133 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:55,134 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,134 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:55,137 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:55,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:55,138 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:55,140 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,140 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:55,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:55,142 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:55,145 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:55,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:55,146 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:55,146 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:55,146 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:55,147 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:55,147 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:55,147 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:55,147 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:55,147 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-21 01:53:55,147 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:55,147 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:55,147 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:55,147 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:55,147 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:55,147 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:55,188 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/785fbb29ac774d1582d0ba5fcf0f0b9c 2023-04-21 01:53:55,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:55,195 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:55,216 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:55,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59276 deadline: 1682042095211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:55,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042095215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:55,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042095217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:55,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59272 deadline: 1682042095217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,242 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/903002e959ab458a837bcdfc33029b95 2023-04-21 01:53:55,280 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/2393e1ab44a54a09b45fc569a6bb0d38 2023-04-21 01:53:55,289 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/785fbb29ac774d1582d0ba5fcf0f0b9c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/785fbb29ac774d1582d0ba5fcf0f0b9c 2023-04-21 01:53:55,297 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/785fbb29ac774d1582d0ba5fcf0f0b9c, entries=150, sequenceid=531, filesize=11.9 K 2023-04-21 01:53:55,298 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/903002e959ab458a837bcdfc33029b95 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/903002e959ab458a837bcdfc33029b95 2023-04-21 01:53:55,305 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/903002e959ab458a837bcdfc33029b95, entries=150, sequenceid=531, filesize=11.9 K 2023-04-21 01:53:55,306 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/2393e1ab44a54a09b45fc569a6bb0d38 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/2393e1ab44a54a09b45fc569a6bb0d38 2023-04-21 01:53:55,312 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/2393e1ab44a54a09b45fc569a6bb0d38, entries=150, sequenceid=531, filesize=11.9 K 2023-04-21 01:53:55,313 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 166ms, sequenceid=531, compaction requested=true 2023-04-21 01:53:55,314 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:55,314 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:55,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:55,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:55,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:55,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:55,316 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,316 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:55,316 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,316 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:55,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:55,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:55,322 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-21 01:53:55,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:55,324 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:55,324 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:55,324 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:55,324 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:55,324 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:55,324 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:55,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:55,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54379392[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:55,326 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:55,326 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,328 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,328 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:55,328 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:55,328 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:55,328 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,328 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:55,332 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:55,332 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:55,332 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:55,332 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:55,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:55,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:55,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:55,337 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:55,337 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:55,337 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:55,337 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:55,337 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:55,341 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,341 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:55,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,341 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:55,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:55,342 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,343 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/04eb6ec9a26e4dcc8db5b734c493a2f7 2023-04-21 01:53:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:55,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:55,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:55,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59276 deadline: 1682042095338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:55,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042095339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,350 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:55,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59272 deadline: 1682042095346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:55,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59306 deadline: 1682042095348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,359 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:55,359 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:55,359 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:55,359 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:55,359 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:55,359 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,359 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,360 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:55,360 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,360 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:55,360 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:55,359 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:55,360 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:55,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:55,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:55,360 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:55,387 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/321d5f27a0664aceb223a84f43cf6444 2023-04-21 01:53:55,411 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/67f97166b43b4d2088c17c29001996bb 2023-04-21 01:53:55,418 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/04eb6ec9a26e4dcc8db5b734c493a2f7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/04eb6ec9a26e4dcc8db5b734c493a2f7 2023-04-21 01:53:55,424 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/04eb6ec9a26e4dcc8db5b734c493a2f7, entries=250, sequenceid=548, filesize=16.7 K 2023-04-21 01:53:55,425 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/321d5f27a0664aceb223a84f43cf6444 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/321d5f27a0664aceb223a84f43cf6444 2023-04-21 01:53:55,431 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/321d5f27a0664aceb223a84f43cf6444, entries=150, sequenceid=548, filesize=11.9 K 2023-04-21 01:53:55,432 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/67f97166b43b4d2088c17c29001996bb as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/67f97166b43b4d2088c17c29001996bb 2023-04-21 01:53:55,441 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/67f97166b43b4d2088c17c29001996bb, entries=150, sequenceid=548, filesize=11.9 K 2023-04-21 01:53:55,443 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 120ms, sequenceid=548, compaction requested=true 2023-04-21 01:53:55,443 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:55,443 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-21 01:53:55,443 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-21 01:53:55,443 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-21 01:53:55,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:55,458 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-21 01:53:55,458 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:55,458 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:55,458 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:55,459 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:55,459 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:55,459 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:55,461 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:55,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:55,464 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:55,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:55,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:55,480 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:55,480 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:55,480 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:55,480 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:55,481 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,481 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:55,482 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,482 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:55,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:55,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:55,483 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,483 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:55,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:55,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:55,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:55,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:55,485 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:55,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:55,485 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:55,485 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:55,485 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:55,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:55,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59272 deadline: 1682042095487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:55,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59306 deadline: 1682042095489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:55,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59276 deadline: 1682042095491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:55,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042095491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,504 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:55,504 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:55,504 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:55,505 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:55,505 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:55,505 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:55,505 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:55,505 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:55,505 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,512 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:55,513 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:55,514 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,514 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:55,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:55,515 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,515 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:55,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:55,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:55,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47c37d48[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:55,518 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:55,518 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,520 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:55,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:55,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:55,520 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,521 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:55,522 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,522 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:55,522 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:55,522 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:55,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:55,526 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:55,526 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:55,526 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:55,526 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:55,526 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:55,528 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,528 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:55,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,529 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:55,531 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:55,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,534 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,539 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/06731cb476b646879256f4cd176ca112 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/06731cb476b646879256f4cd176ca112 2023-04-21 01:53:55,541 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=565 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/b6c1eef51bf548bb8db8baf2edd23e88 2023-04-21 01:53:55,542 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:55,542 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:55,543 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,543 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:55,543 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:55,543 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:55,543 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:55,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:55,543 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:55,543 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:55,543 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:55,543 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,543 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,543 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,543 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:55,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:55,555 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/cc246645a35449a7a9e27cf9e0e56fec as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/cc246645a35449a7a9e27cf9e0e56fec 2023-04-21 01:53:55,564 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/B of 16b5c71a3c47da32fe9f26ec9e41dfb8 into 06731cb476b646879256f4cd176ca112(size=12.5 K), total size for store is 36.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:55,564 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:55,564 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/B, priority=12, startTime=1682042035076; duration=0sec 2023-04-21 01:53:55,565 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-21 01:53:55,565 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 4 compacting, 2 eligible, 16 blocking 2023-04-21 01:53:55,568 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-21 01:53:55,568 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-21 01:53:55,568 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. because compaction request was cancelled 2023-04-21 01:53:55,568 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-21 01:53:55,574 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73628 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-21 01:53:55,574 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/C is initiating minor compaction (all files) 2023-04-21 01:53:55,574 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/C in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:55,574 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/9404233302cd4638998c17b3a5260dad, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/f26105d4af68439ba4bcebacb3a886ce, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/78542f67720249dcb31e3c798ded7510, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/8e387eee7fd8445cb664c04937f1d3a7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/2393e1ab44a54a09b45fc569a6bb0d38, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/67f97166b43b4d2088c17c29001996bb] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=71.9 K 2023-04-21 01:53:55,575 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 9404233302cd4638998c17b3a5260dad, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1682042033082 2023-04-21 01:53:55,575 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting f26105d4af68439ba4bcebacb3a886ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682042033218 2023-04-21 01:53:55,576 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 78542f67720249dcb31e3c798ded7510, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1682042033268 2023-04-21 01:53:55,577 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 8e387eee7fd8445cb664c04937f1d3a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1682042034450 2023-04-21 01:53:55,579 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 2393e1ab44a54a09b45fc569a6bb0d38, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1682042034558 2023-04-21 01:53:55,579 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/A of 16b5c71a3c47da32fe9f26ec9e41dfb8 into cc246645a35449a7a9e27cf9e0e56fec(size=12.5 K), total size for store is 41.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:55,579 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:55,579 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/A, priority=12, startTime=1682042035076; duration=0sec 2023-04-21 01:53:55,579 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-21 01:53:55,579 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 6 compacting, 0 eligible, 16 blocking 2023-04-21 01:53:55,579 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-21 01:53:55,579 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-21 01:53:55,580 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. because compaction request was cancelled 2023-04-21 01:53:55,580 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:53:55,580 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 67f97166b43b4d2088c17c29001996bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1682042035215 2023-04-21 01:53:55,582 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:53:55,583 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/B is initiating minor compaction (all files) 2023-04-21 01:53:55,583 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/B in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:55,583 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/06731cb476b646879256f4cd176ca112, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/903002e959ab458a837bcdfc33029b95, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/321d5f27a0664aceb223a84f43cf6444] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=36.3 K 2023-04-21 01:53:55,584 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 06731cb476b646879256f4cd176ca112, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1682042034450 2023-04-21 01:53:55,584 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 903002e959ab458a837bcdfc33029b95, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1682042034558 2023-04-21 01:53:55,585 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 321d5f27a0664aceb223a84f43cf6444, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1682042035215 2023-04-21 01:53:55,609 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#C#compaction#67 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:55,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:55,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59306 deadline: 1682042095599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:55,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59272 deadline: 1682042095611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:55,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59276 deadline: 1682042095622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:55,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042095623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,638 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=565 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/0b3d26375ce14c4b9f4257c270e08119 2023-04-21 01:53:55,639 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#B#compaction#68 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:55,651 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:55,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:55,655 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:55,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:55,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:55,666 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:55,666 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:55,666 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:55,666 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:55,673 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,673 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:55,674 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:55,674 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:55,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:55,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:55,679 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,679 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:55,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:55,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:55,700 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:55,700 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:55,700 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:55,704 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:55,704 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:55,704 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:55,704 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:55,704 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:55,704 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:55,704 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:55,704 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:55,704 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,705 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:55,705 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:55,706 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,707 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:55,707 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:55,707 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:55,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:55,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b9ca5ec[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:55,709 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:55,709 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,712 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,712 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:55,713 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:55,713 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:55,713 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,713 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:55,715 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,715 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:55,715 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:55,715 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:55,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:55,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:55,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:55,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:55,720 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:55,720 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:55,720 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:55,720 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:55,720 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:55,722 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,722 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,722 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:55,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:55,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:55,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:55,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:55,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:55,723 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,725 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,728 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:55,728 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:55,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:55,728 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:55,728 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:55,728 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:55,728 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:55,728 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:55,728 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:55,729 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,729 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,729 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:55,729 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:55,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:55,729 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,729 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:55,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:55,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59306 deadline: 1682042095817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:55,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59272 deadline: 1682042095818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:55,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59276 deadline: 1682042095827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,829 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:55,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:55,833 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:55,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:55,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042095831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:55,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:55,844 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:55,844 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:55,844 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:55,844 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:55,845 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,845 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:55,847 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,847 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:55,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:55,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:55,848 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,848 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:55,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:55,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:55,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:55,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:55,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:55,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:55,852 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:55,852 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:55,852 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:55,866 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:55,866 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:55,879 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:55,879 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:55,879 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:55,879 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:55,879 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:55,879 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:55,879 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,879 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:55,879 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:55,882 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,882 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:55,883 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,883 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:55,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:55,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18204211[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:55,885 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:55,885 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,888 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,888 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:55,888 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:55,888 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:55,889 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,890 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:55,891 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,891 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:55,891 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:55,891 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:55,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:55,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:55,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:55,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:55,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:55,898 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:55,898 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:55,898 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:55,898 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:55,898 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:55,901 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,901 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:55,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,901 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:55,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,903 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:55,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,905 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:55,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,918 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,918 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:55,918 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:55,918 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:55,918 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:55,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:55,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:55,918 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:55,919 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,919 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:55,919 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:55,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:55,918 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:55,919 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:55,919 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:55,919 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:55,919 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:55,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:56,019 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:56,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:56,023 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:56,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:56,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:56,024 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:56,024 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:56,024 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:56,024 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:56,025 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,025 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:56,026 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,026 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:56,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:56,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:56,027 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,027 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:56,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:56,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:56,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:56,028 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:56,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:56,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:56,028 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:56,028 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:56,028 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:56,029 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:56,029 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:56,030 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:56,030 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:56,030 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:56,030 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:56,030 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:56,030 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:56,030 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,030 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:56,030 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:56,032 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,032 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:56,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:56,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:56,033 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,033 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:56,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:56,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:56,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7063e984[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:56,036 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:56,036 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,038 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,039 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:56,039 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:56,039 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:56,039 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,039 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:56,040 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:56,040 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:56,040 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,040 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:56,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:56,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:56,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:56,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:56,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:56,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:56,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:56,045 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:56,045 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:56,045 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:56,045 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:56,045 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:56,047 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:56,048 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,048 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:56,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:56,049 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:56,050 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,061 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:56,061 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,061 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:56,061 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:56,061 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:56,061 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:56,061 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:56,061 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:56,062 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:56,062 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:56,062 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:56,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:56,061 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,064 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,064 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:56,084 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/5a31cf5e56a64e62a72dffb0745b6aa9 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/5a31cf5e56a64e62a72dffb0745b6aa9 2023-04-21 01:53:56,091 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/C of 16b5c71a3c47da32fe9f26ec9e41dfb8 into 5a31cf5e56a64e62a72dffb0745b6aa9(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:56,091 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:56,091 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/C, priority=10, startTime=1682042035443; duration=0sec 2023-04-21 01:53:56,091 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:56,104 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/a043d0e57a964a42b4cbefa8454c5bb1 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/a043d0e57a964a42b4cbefa8454c5bb1 2023-04-21 01:53:56,107 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=565 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/1adef58ece594d42adde12f062976884 2023-04-21 01:53:56,111 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/B of 16b5c71a3c47da32fe9f26ec9e41dfb8 into a043d0e57a964a42b4cbefa8454c5bb1(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:56,111 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:56,111 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/B, priority=13, startTime=1682042035443; duration=0sec 2023-04-21 01:53:56,111 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:56,114 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/b6c1eef51bf548bb8db8baf2edd23e88 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b6c1eef51bf548bb8db8baf2edd23e88 2023-04-21 01:53:56,122 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b6c1eef51bf548bb8db8baf2edd23e88, entries=250, sequenceid=565, filesize=16.7 K 2023-04-21 01:53:56,123 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/0b3d26375ce14c4b9f4257c270e08119 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/0b3d26375ce14c4b9f4257c270e08119 2023-04-21 01:53:56,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:56,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59306 deadline: 1682042096124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:56,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59272 deadline: 1682042096124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,130 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/0b3d26375ce14c4b9f4257c270e08119, entries=150, sequenceid=565, filesize=11.9 K 2023-04-21 01:53:56,131 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/1adef58ece594d42adde12f062976884 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/1adef58ece594d42adde12f062976884 2023-04-21 01:53:56,133 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:56,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59276 deadline: 1682042096130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,136 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/1adef58ece594d42adde12f062976884, entries=150, sequenceid=565, filesize=11.9 K 2023-04-21 01:53:56,137 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 679ms, sequenceid=565, compaction requested=true 2023-04-21 01:53:56,137 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:56,137 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:56,137 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:53:56,137 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-21 01:53:56,137 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:53:56,138 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:53:56,138 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-21 01:53:56,138 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-21 01:53:56,138 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. because compaction request was cancelled 2023-04-21 01:53:56,139 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-21 01:53:56,139 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 59152 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:53:56,139 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/A is initiating minor compaction (all files) 2023-04-21 01:53:56,139 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/A in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:56,139 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/cc246645a35449a7a9e27cf9e0e56fec, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/785fbb29ac774d1582d0ba5fcf0f0b9c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/04eb6ec9a26e4dcc8db5b734c493a2f7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b6c1eef51bf548bb8db8baf2edd23e88] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=57.8 K 2023-04-21 01:53:56,139 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-21 01:53:56,139 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-21 01:53:56,140 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. because compaction request was cancelled 2023-04-21 01:53:56,140 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting cc246645a35449a7a9e27cf9e0e56fec, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1682042034450 2023-04-21 01:53:56,140 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 785fbb29ac774d1582d0ba5fcf0f0b9c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1682042034558 2023-04-21 01:53:56,141 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 04eb6ec9a26e4dcc8db5b734c493a2f7, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1682042035210 2023-04-21 01:53:56,141 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting b6c1eef51bf548bb8db8baf2edd23e88, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1682042035335 2023-04-21 01:53:56,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:56,143 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-21 01:53:56,143 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:56,143 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:56,143 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:56,143 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:56,143 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:56,143 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:56,154 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/77557505b95144e68b918440756fead2 2023-04-21 01:53:56,156 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#A#compaction#71 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:56,162 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:56,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:56,167 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:56,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:56,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:56,177 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:56,177 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:56,177 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:56,177 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:56,179 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,180 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:56,181 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:56,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:56,182 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:56,182 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,182 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:56,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:56,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:56,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042096187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:56,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:56,207 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:56,207 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:56,207 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:56,208 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:56,208 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:56,208 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:56,208 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:56,209 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:56,210 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:56,210 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:56,210 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:56,210 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,218 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:56,218 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:56,221 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:56,221 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:56,223 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:56,223 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:56,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:56,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c16e3fd[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:56,225 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:56,225 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,226 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/f356b223339d44bb88c85ea96a6d5e9c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/f356b223339d44bb88c85ea96a6d5e9c 2023-04-21 01:53:56,227 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,227 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:56,228 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:56,228 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:56,228 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,228 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:56,237 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/A of 16b5c71a3c47da32fe9f26ec9e41dfb8 into f356b223339d44bb88c85ea96a6d5e9c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:56,237 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:56,238 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/A, priority=12, startTime=1682042036137; duration=0sec 2023-04-21 01:53:56,238 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:56,255 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/01ecaebb7f104f079d1e6b021f25fa36 2023-04-21 01:53:56,279 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/a57ce6ca99c140c4a0a9fb064e9bc739 2023-04-21 01:53:56,285 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/77557505b95144e68b918440756fead2 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/77557505b95144e68b918440756fead2 2023-04-21 01:53:56,290 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:56,290 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,290 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:56,290 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:56,290 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/77557505b95144e68b918440756fead2, entries=150, sequenceid=586, filesize=11.9 K 2023-04-21 01:53:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:56,292 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/01ecaebb7f104f079d1e6b021f25fa36 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/01ecaebb7f104f079d1e6b021f25fa36 2023-04-21 01:53:56,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:56,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:56,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:56,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:56,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:56,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042096291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,299 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/01ecaebb7f104f079d1e6b021f25fa36, entries=150, sequenceid=586, filesize=11.9 K 2023-04-21 01:53:56,300 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/a57ce6ca99c140c4a0a9fb064e9bc739 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/a57ce6ca99c140c4a0a9fb064e9bc739 2023-04-21 01:53:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:56,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:56,304 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:56,304 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:56,304 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:56,304 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:56,304 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:56,308 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/a57ce6ca99c140c4a0a9fb064e9bc739, entries=150, sequenceid=586, filesize=11.9 K 2023-04-21 01:53:56,311 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 168ms, sequenceid=586, compaction requested=true 2023-04-21 01:53:56,311 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:56,311 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:56,311 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-21 01:53:56,311 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:53:56,311 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:53:56,311 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:53:56,312 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-21 01:53:56,312 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-21 01:53:56,312 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. because compaction request was cancelled 2023-04-21 01:53:56,312 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:53:56,313 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:53:56,313 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/B is initiating minor compaction (all files) 2023-04-21 01:53:56,313 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/B in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:56,313 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/a043d0e57a964a42b4cbefa8454c5bb1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/0b3d26375ce14c4b9f4257c270e08119, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/01ecaebb7f104f079d1e6b021f25fa36] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=36.4 K 2023-04-21 01:53:56,314 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:53:56,314 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/C is initiating minor compaction (all files) 2023-04-21 01:53:56,314 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/C in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:56,314 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/5a31cf5e56a64e62a72dffb0745b6aa9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/1adef58ece594d42adde12f062976884, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/a57ce6ca99c140c4a0a9fb064e9bc739] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=36.4 K 2023-04-21 01:53:56,314 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting a043d0e57a964a42b4cbefa8454c5bb1, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1682042035215 2023-04-21 01:53:56,315 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 5a31cf5e56a64e62a72dffb0745b6aa9, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1682042035215 2023-04-21 01:53:56,315 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 0b3d26375ce14c4b9f4257c270e08119, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1682042035346 2023-04-21 01:53:56,316 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 1adef58ece594d42adde12f062976884, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1682042035346 2023-04-21 01:53:56,316 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 01ecaebb7f104f079d1e6b021f25fa36, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1682042035489 2023-04-21 01:53:56,317 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting a57ce6ca99c140c4a0a9fb064e9bc739, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1682042035489 2023-04-21 01:53:56,327 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#B#compaction#74 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:56,332 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#C#compaction#75 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:56,360 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/bae4038ed4bd4d99a16c3538ea3f526b as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/bae4038ed4bd4d99a16c3538ea3f526b 2023-04-21 01:53:56,369 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/C of 16b5c71a3c47da32fe9f26ec9e41dfb8 into bae4038ed4bd4d99a16c3538ea3f526b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:56,369 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:56,369 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/C, priority=13, startTime=1682042036311; duration=0sec 2023-04-21 01:53:56,369 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:56,382 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,382 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:56,382 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:56,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:56,384 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:56,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:56,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:56,386 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,437 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,438 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:56,438 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:56,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:56,438 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:56,438 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:56,438 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:56,438 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:56,438 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:56,438 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,438 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,438 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:56,438 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,438 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:56,438 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:56,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:56,503 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-21 01:53:56,503 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:56,503 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:56,503 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:56,503 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:56,503 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:56,504 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:56,514 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/3b7ebdf308fa48a2acdd6bab98605adf 2023-04-21 01:53:56,539 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:56,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:56,542 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:56,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:56,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:56,544 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:56,544 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:56,544 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:56,544 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:56,544 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,544 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:56,546 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/52cf492a4a6048ddb59c7c3425db752e 2023-04-21 01:53:56,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:56,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042096556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,598 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:56,598 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:56,598 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:56,598 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:56,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:56,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:56,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:56,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:56,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:56,621 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:56,621 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:56,621 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:56,628 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:56,628 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:56,628 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:56,628 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:56,628 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:56,628 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:56,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:56,630 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:56,632 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:56,632 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,632 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:56,632 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:56,632 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/b5ba3d6b5f9b44e7ad5fc46527804c21 2023-04-21 01:53:56,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59272 deadline: 1682042096629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:56,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59306 deadline: 1682042096633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,642 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/3b7ebdf308fa48a2acdd6bab98605adf as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/3b7ebdf308fa48a2acdd6bab98605adf 2023-04-21 01:53:56,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:56,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59276 deadline: 1682042096638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,655 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/3b7ebdf308fa48a2acdd6bab98605adf, entries=150, sequenceid=604, filesize=11.9 K 2023-04-21 01:53:56,656 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/52cf492a4a6048ddb59c7c3425db752e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/52cf492a4a6048ddb59c7c3425db752e 2023-04-21 01:53:56,667 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,667 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:56,667 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/52cf492a4a6048ddb59c7c3425db752e, entries=150, sequenceid=604, filesize=11.9 K 2023-04-21 01:53:56,668 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,668 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:56,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:56,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:56,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:56,669 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/b5ba3d6b5f9b44e7ad5fc46527804c21 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/b5ba3d6b5f9b44e7ad5fc46527804c21 2023-04-21 01:53:56,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:56,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:56,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@121b8056[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:56,670 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:56,671 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,674 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/b5ba3d6b5f9b44e7ad5fc46527804c21, entries=150, sequenceid=604, filesize=11.9 K 2023-04-21 01:53:56,675 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 172ms, sequenceid=604, compaction requested=true 2023-04-21 01:53:56,675 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:56,675 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:56,675 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:53:56,675 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:53:56,676 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-21 01:53:56,677 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:53:56,677 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/A is initiating minor compaction (all files) 2023-04-21 01:53:56,677 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/A in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:56,677 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/f356b223339d44bb88c85ea96a6d5e9c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/77557505b95144e68b918440756fead2, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/3b7ebdf308fa48a2acdd6bab98605adf] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=36.5 K 2023-04-21 01:53:56,678 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting f356b223339d44bb88c85ea96a6d5e9c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1682042035346 2023-04-21 01:53:56,678 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 77557505b95144e68b918440756fead2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1682042035489 2023-04-21 01:53:56,680 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 3b7ebdf308fa48a2acdd6bab98605adf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1682042036152 2023-04-21 01:53:56,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:56,692 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-21 01:53:56,694 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:56,695 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:56,695 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:56,695 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:56,695 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:56,695 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:56,695 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#A#compaction#79 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:56,711 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,711 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:56,711 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:56,711 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:56,712 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,712 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:56,744 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,744 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:56,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:56,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:56,744 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:56,744 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:56,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:56,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:56,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:56,749 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:56,749 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:56,749 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:56,749 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:56,749 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:56,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:56,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042096753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,768 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/d0fe3d88c2df4f9e9d9af3b41e811848 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/d0fe3d88c2df4f9e9d9af3b41e811848 2023-04-21 01:53:56,779 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/B of 16b5c71a3c47da32fe9f26ec9e41dfb8 into d0fe3d88c2df4f9e9d9af3b41e811848(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:56,779 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:56,779 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/B, priority=13, startTime=1682042036311; duration=0sec 2023-04-21 01:53:56,779 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-21 01:53:56,779 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-21 01:53:56,780 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-21 01:53:56,781 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-21 01:53:56,781 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. because compaction request was cancelled 2023-04-21 01:53:56,781 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-21 01:53:56,782 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-21 01:53:56,782 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-21 01:53:56,782 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. because compaction request was cancelled 2023-04-21 01:53:56,806 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,806 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:56,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:56,806 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:56,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:56,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:56,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:56,807 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:56,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:56,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:56,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,809 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:56,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,852 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,852 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:56,852 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:56,852 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,852 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,852 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:56,852 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:56,852 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,852 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:56,852 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:56,852 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:56,852 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:56,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:56,852 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:56,852 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:56,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:56,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:56,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042096856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:56,953 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:56,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:56,956 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:56,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:56,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:56,957 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:56,957 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:56,957 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:56,957 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:56,958 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:56,958 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:57,021 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:57,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:57,021 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:57,022 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,022 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:57,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:57,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:57,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:57,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:57,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:57,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:57,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:57,028 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:57,028 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:57,028 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:57,036 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:57,036 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:57,036 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:57,036 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:57,036 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:57,036 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:57,036 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:57,036 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:57,036 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,036 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:57,036 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:57,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:57,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042097060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,090 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,090 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:57,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:57,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:57,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:57,091 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:57,091 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:57,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:57,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:57,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:57,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:57,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@83178ce[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:57,094 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:57,094 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:57,146 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/ce471700f59848e6bfc34596cec60432 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/ce471700f59848e6bfc34596cec60432 2023-04-21 01:53:57,152 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:57,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:57,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:57,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:57,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:57,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:57,153 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,153 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:57,154 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/A of 16b5c71a3c47da32fe9f26ec9e41dfb8 into ce471700f59848e6bfc34596cec60432(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:57,154 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:57,154 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/A, priority=13, startTime=1682042036675; duration=0sec 2023-04-21 01:53:57,154 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:57,169 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=624 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/576f481dfad24e06bdc41efedf5fc28f 2023-04-21 01:53:57,181 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=624 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/e714ca69d7e14c0a899cf95a4029cfd7 2023-04-21 01:53:57,194 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=624 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/d9e11af48b9547d69afbaec8029b5244 2023-04-21 01:53:57,199 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/576f481dfad24e06bdc41efedf5fc28f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/576f481dfad24e06bdc41efedf5fc28f 2023-04-21 01:53:57,206 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/576f481dfad24e06bdc41efedf5fc28f, entries=200, sequenceid=624, filesize=14.3 K 2023-04-21 01:53:57,207 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/e714ca69d7e14c0a899cf95a4029cfd7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/e714ca69d7e14c0a899cf95a4029cfd7 2023-04-21 01:53:57,212 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/e714ca69d7e14c0a899cf95a4029cfd7, entries=150, sequenceid=624, filesize=11.9 K 2023-04-21 01:53:57,213 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/d9e11af48b9547d69afbaec8029b5244 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/d9e11af48b9547d69afbaec8029b5244 2023-04-21 01:53:57,213 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:57,213 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,213 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:57,213 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:57,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:57,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:57,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:57,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:57,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:57,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:57,217 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:57,217 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:57,217 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:57,217 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:57,217 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:57,236 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/d9e11af48b9547d69afbaec8029b5244, entries=150, sequenceid=624, filesize=11.9 K 2023-04-21 01:53:57,238 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 546ms, sequenceid=624, compaction requested=true 2023-04-21 01:53:57,238 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:57,238 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:57,238 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-21 01:53:57,238 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:53:57,238 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-21 01:53:57,238 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:53:57,239 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-21 01:53:57,239 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-21 01:53:57,239 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. because compaction request was cancelled 2023-04-21 01:53:57,239 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:53:57,239 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:53:57,240 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/B is initiating minor compaction (all files) 2023-04-21 01:53:57,240 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/B in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:57,240 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/d0fe3d88c2df4f9e9d9af3b41e811848, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/52cf492a4a6048ddb59c7c3425db752e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/e714ca69d7e14c0a899cf95a4029cfd7] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=36.5 K 2023-04-21 01:53:57,240 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting d0fe3d88c2df4f9e9d9af3b41e811848, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1682042035489 2023-04-21 01:53:57,241 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:53:57,241 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/C is initiating minor compaction (all files) 2023-04-21 01:53:57,241 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/C in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:57,241 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/bae4038ed4bd4d99a16c3538ea3f526b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/b5ba3d6b5f9b44e7ad5fc46527804c21, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/d9e11af48b9547d69afbaec8029b5244] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=36.5 K 2023-04-21 01:53:57,241 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 52cf492a4a6048ddb59c7c3425db752e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1682042036152 2023-04-21 01:53:57,242 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting bae4038ed4bd4d99a16c3538ea3f526b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1682042035489 2023-04-21 01:53:57,242 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting e714ca69d7e14c0a899cf95a4029cfd7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=624, earliestPutTs=1682042036682 2023-04-21 01:53:57,242 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting b5ba3d6b5f9b44e7ad5fc46527804c21, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1682042036152 2023-04-21 01:53:57,242 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting d9e11af48b9547d69afbaec8029b5244, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=624, earliestPutTs=1682042036682 2023-04-21 01:53:57,250 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#B#compaction#83 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:57,253 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#C#compaction#84 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:57,264 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/8f153e92b56d458a8fe901886fa79703 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/8f153e92b56d458a8fe901886fa79703 2023-04-21 01:53:57,280 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/4fad11ba83554c0f908962810a493969 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/4fad11ba83554c0f908962810a493969 2023-04-21 01:53:57,280 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/B of 16b5c71a3c47da32fe9f26ec9e41dfb8 into 8f153e92b56d458a8fe901886fa79703(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:57,280 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:57,280 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/B, priority=13, startTime=1682042037238; duration=0sec 2023-04-21 01:53:57,280 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:57,282 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:57,282 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:57,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:57,282 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:57,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:57,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:57,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:57,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:57,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:57,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:57,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:57,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:57,284 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:57,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:57,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:57,287 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,289 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/C of 16b5c71a3c47da32fe9f26ec9e41dfb8 into 4fad11ba83554c0f908962810a493969(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:57,289 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:57,289 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/C, priority=13, startTime=1682042037238; duration=0sec 2023-04-21 01:53:57,289 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:57,344 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,344 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:57,344 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:57,344 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,344 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:57,345 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:57,345 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:57,344 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:57,345 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:57,345 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:57,345 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:57,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:57,345 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:57,345 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:57,345 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:57,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:57,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:57,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:57,376 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-21 01:53:57,376 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:57,377 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:57,377 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:57,377 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:57,377 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:57,377 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:57,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:57,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042097425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,445 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:57,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:57,448 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:57,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:57,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:57,450 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:57,450 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:57,450 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:57,450 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:57,450 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:57,450 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:57,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:57,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042097527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,621 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:57,621 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:57,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:57,621 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:57,621 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:57,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:57,622 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:57,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:57,623 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:57,623 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:57,623 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:57,625 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:57,625 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:57,625 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:57,625 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:57,626 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:57,626 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:57,626 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:57,626 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:57,626 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,626 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:57,626 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:57,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:57,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59272 deadline: 1682042097641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,644 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,644 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:57,645 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:57,645 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:57,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:57,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:57,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:57,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:57,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59306 deadline: 1682042097645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:57,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:57,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67f19f85[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:57,648 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:57,648 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:57,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:57,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59276 deadline: 1682042097649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,682 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:57,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:57,683 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:57,683 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:57,683 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:57,683 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,683 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:57,705 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:57,706 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:57,706 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:57,705 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:57,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:57,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:57,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:57,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:57,710 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:57,710 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:57,710 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:57,710 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:57,710 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:57,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:57,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042097733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,751 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:57,751 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:57,752 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:57,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:57,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:57,752 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:57,756 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:57,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:57,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,811 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/6cc0e4ec58724f5e8e1e81abbbbfd628 2023-04-21 01:53:57,821 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:57,821 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:57,821 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:57,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:57,821 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:57,821 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:57,821 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:57,821 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,821 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:57,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:57,821 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:57,821 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:57,821 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:57,822 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:57,824 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:57,824 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:57,825 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/22591c4b86844ea6916884d5334b9e1e 2023-04-21 01:53:57,924 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:57,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:57,927 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:57,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:57,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:57,928 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:57,928 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:57,928 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:57,928 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:57,929 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:57,929 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:57,982 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,982 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:57,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:57,983 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,983 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:57,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:57,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:57,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:57,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:57,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:57,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:57,984 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:57,984 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:57,984 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:57,984 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:57,984 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:57,984 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:57,984 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:57,984 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:57,985 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:57,985 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:57,985 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:57,985 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:57,985 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:57,985 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:58,036 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:58,037 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,037 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:58,037 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:58,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:58,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:58,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:58,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:58,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ce3818d[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:58,040 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:58,040 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:58,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:58,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042098039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,113 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:58,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:58,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:58,114 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:58,114 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:58,114 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:58,175 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,175 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:58,175 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:58,175 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:58,175 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:58,175 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:58,175 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:58,175 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:58,176 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:58,176 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:58,176 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:58,245 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/dd7e53da8a05472884294e290ce259ac 2023-04-21 01:53:58,247 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:58,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:58,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:58,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:58,248 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:58,248 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:58,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:58,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:58,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:58,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:58,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:58,249 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:58,253 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:58,255 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/6cc0e4ec58724f5e8e1e81abbbbfd628 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/6cc0e4ec58724f5e8e1e81abbbbfd628 2023-04-21 01:53:58,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,264 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/6cc0e4ec58724f5e8e1e81abbbbfd628, entries=150, sequenceid=641, filesize=11.9 K 2023-04-21 01:53:58,267 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/22591c4b86844ea6916884d5334b9e1e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/22591c4b86844ea6916884d5334b9e1e 2023-04-21 01:53:58,274 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/22591c4b86844ea6916884d5334b9e1e, entries=150, sequenceid=641, filesize=11.9 K 2023-04-21 01:53:58,275 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,275 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:58,275 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:58,276 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/dd7e53da8a05472884294e290ce259ac as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/dd7e53da8a05472884294e290ce259ac 2023-04-21 01:53:58,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:58,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:58,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:58,275 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:58,275 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:58,276 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:58,275 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:58,276 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:58,276 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,276 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:58,276 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:58,276 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:58,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:58,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:58,276 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:58,277 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:58,281 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/dd7e53da8a05472884294e290ce259ac, entries=150, sequenceid=641, filesize=11.9 K 2023-04-21 01:53:58,282 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 906ms, sequenceid=641, compaction requested=true 2023-04-21 01:53:58,282 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:58,282 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:58,282 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:53:58,283 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-21 01:53:58,283 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:58,283 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:53:58,283 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-21 01:53:58,284 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-21 01:53:58,284 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. because compaction request was cancelled 2023-04-21 01:53:58,284 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-21 01:53:58,284 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39884 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:53:58,284 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/A is initiating minor compaction (all files) 2023-04-21 01:53:58,284 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/A in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:58,284 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/ce471700f59848e6bfc34596cec60432, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/576f481dfad24e06bdc41efedf5fc28f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/6cc0e4ec58724f5e8e1e81abbbbfd628] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=38.9 K 2023-04-21 01:53:58,286 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-21 01:53:58,286 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-21 01:53:58,286 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting ce471700f59848e6bfc34596cec60432, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1682042036152 2023-04-21 01:53:58,286 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. because compaction request was cancelled 2023-04-21 01:53:58,288 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 576f481dfad24e06bdc41efedf5fc28f, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=624, earliestPutTs=1682042036550 2023-04-21 01:53:58,288 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 6cc0e4ec58724f5e8e1e81abbbbfd628, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1682042036740 2023-04-21 01:53:58,297 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#A#compaction#88 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:58,377 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:58,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:58,379 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:58,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:58,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:58,380 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:58,381 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:58,381 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:58,381 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:58,381 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:58,381 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:58,413 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:58,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:58,416 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:58,416 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,416 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:58,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:58,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:58,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:58,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:58,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:58,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:58,418 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:58,418 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:58,418 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:58,420 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:58,420 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:58,420 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:58,420 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:58,420 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-21 01:53:58,420 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:58,421 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:58,421 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:58,421 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:58,421 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:58,421 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:58,434 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=660 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/faa8d302554b496f9a64ca2dc90b05f9 2023-04-21 01:53:58,458 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=660 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/0217985618e346fca914f671a8e52296 2023-04-21 01:53:58,474 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=660 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/9a865824f87b4133a05cee5cdfca8921 2023-04-21 01:53:58,480 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/faa8d302554b496f9a64ca2dc90b05f9 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/faa8d302554b496f9a64ca2dc90b05f9 2023-04-21 01:53:58,486 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/faa8d302554b496f9a64ca2dc90b05f9, entries=150, sequenceid=660, filesize=11.9 K 2023-04-21 01:53:58,487 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/0217985618e346fca914f671a8e52296 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/0217985618e346fca914f671a8e52296 2023-04-21 01:53:58,493 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/0217985618e346fca914f671a8e52296, entries=150, sequenceid=660, filesize=11.9 K 2023-04-21 01:53:58,495 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/9a865824f87b4133a05cee5cdfca8921 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/9a865824f87b4133a05cee5cdfca8921 2023-04-21 01:53:58,501 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/9a865824f87b4133a05cee5cdfca8921, entries=150, sequenceid=660, filesize=11.9 K 2023-04-21 01:53:58,501 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 81ms, sequenceid=660, compaction requested=true 2023-04-21 01:53:58,502 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:58,502 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:58,544 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:58,544 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:58,545 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:58,545 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:58,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:58,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:58,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3234b777[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:58,547 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:58,547 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:58,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:58,558 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-21 01:53:58,558 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:58,558 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:58,558 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:58,558 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:58,558 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:58,558 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:58,606 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:58,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:58,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:58,607 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:58,607 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:58,607 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:58,607 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,607 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:58,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682042038586, max timestamp=1682042038609Num uniques 859; 2023-04-21 01:53:58,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-21 01:53:58,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682042038586, max timestamp=1682042038609Num uniques 859; 2023-04-21 01:53:58,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682042038586, max timestamp=1682042038609Num uniques 859; 2023-04-21 01:53:58,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-21 01:53:58,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-21 01:53:58,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:58,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042098639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:58,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042098640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,659 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,659 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:58,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,659 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:58,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:58,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:58,660 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:58,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:58,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:58,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:58,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:58,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:58,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:58,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:58,662 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:58,663 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:58,663 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:58,663 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:58,663 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:58,714 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/058465b805c84962aa48f55fc2045356 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/058465b805c84962aa48f55fc2045356 2023-04-21 01:53:58,720 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/A of 16b5c71a3c47da32fe9f26ec9e41dfb8 into 058465b805c84962aa48f55fc2045356(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:58,720 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:58,720 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/A, priority=13, startTime=1682042038282; duration=0sec 2023-04-21 01:53:58,720 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:58,744 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:58,744 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:58,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:58,744 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:58,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:58,745 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:58,747 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:58,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042098744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:58,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59292 deadline: 1682042098744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,813 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,813 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:58,813 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:58,813 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,813 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:58,813 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:58,813 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:58,813 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:58,813 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:58,813 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:58,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:58,813 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:58,814 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:58,814 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:58,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:58,814 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:58,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:58,914 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:58,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:58,917 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:58,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:58,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:58,919 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:58,919 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:58,919 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:58,919 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:58,919 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:58,919 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:58,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:58,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042098951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:58,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042098952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,967 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/371dff0409574db2a0dd82d9ccb99c8e 2023-04-21 01:53:58,990 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,990 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:58,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:58,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:58,990 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:58,991 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:58,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:58,993 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:58,994 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:58,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:58,994 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:58,994 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:58,995 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:59,003 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:59,003 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:59,003 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/4a98698cedd84d9b9d89bb184450bd3f 2023-04-21 01:53:59,003 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:59,003 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:59,004 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,004 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:59,004 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:59,037 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/6aa9e03a9a5b4e5c894ef500de98e268 2023-04-21 01:53:59,047 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/371dff0409574db2a0dd82d9ccb99c8e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/371dff0409574db2a0dd82d9ccb99c8e 2023-04-21 01:53:59,053 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/371dff0409574db2a0dd82d9ccb99c8e, entries=150, sequenceid=670, filesize=11.9 K 2023-04-21 01:53:59,055 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/4a98698cedd84d9b9d89bb184450bd3f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/4a98698cedd84d9b9d89bb184450bd3f 2023-04-21 01:53:59,067 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:59,067 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,069 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/4a98698cedd84d9b9d89bb184450bd3f, entries=150, sequenceid=670, filesize=11.9 K 2023-04-21 01:53:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:59,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:59,070 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:59,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31404ef7[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:59,070 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:59,070 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/6aa9e03a9a5b4e5c894ef500de98e268 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/6aa9e03a9a5b4e5c894ef500de98e268 2023-04-21 01:53:59,135 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/6aa9e03a9a5b4e5c894ef500de98e268, entries=150, sequenceid=670, filesize=11.9 K 2023-04-21 01:53:59,136 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 578ms, sequenceid=670, compaction requested=true 2023-04-21 01:53:59,136 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:59,136 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:59,136 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:53:59,136 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:53:59,136 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:53:59,137 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:53:59,138 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:53:59,138 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/A is initiating minor compaction (all files) 2023-04-21 01:53:59,138 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/A in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:59,138 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/058465b805c84962aa48f55fc2045356, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/faa8d302554b496f9a64ca2dc90b05f9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/371dff0409574db2a0dd82d9ccb99c8e] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=36.7 K 2023-04-21 01:53:59,140 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 058465b805c84962aa48f55fc2045356, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1682042036740 2023-04-21 01:53:59,140 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting faa8d302554b496f9a64ca2dc90b05f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=660, earliestPutTs=1682042037420 2023-04-21 01:53:59,140 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:53:59,141 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/B is initiating minor compaction (all files) 2023-04-21 01:53:59,141 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/B in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:59,141 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/8f153e92b56d458a8fe901886fa79703, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/22591c4b86844ea6916884d5334b9e1e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/0217985618e346fca914f671a8e52296, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/4a98698cedd84d9b9d89bb184450bd3f] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=48.5 K 2023-04-21 01:53:59,141 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 371dff0409574db2a0dd82d9ccb99c8e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1682042038543 2023-04-21 01:53:59,143 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 8f153e92b56d458a8fe901886fa79703, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=624, earliestPutTs=1682042036682 2023-04-21 01:53:59,143 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 22591c4b86844ea6916884d5334b9e1e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1682042036740 2023-04-21 01:53:59,144 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 0217985618e346fca914f671a8e52296, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=660, earliestPutTs=1682042037420 2023-04-21 01:53:59,145 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 4a98698cedd84d9b9d89bb184450bd3f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1682042038543 2023-04-21 01:53:59,154 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#A#compaction#95 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:59,166 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#B#compaction#96 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:59,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:59,277 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-04-21 01:53:59,278 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:59,278 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:59,278 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:59,278 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:59,278 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:59,278 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:53:59,278 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:59,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:59,278 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:59,278 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:59,278 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:59,279 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,279 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:59,292 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=708 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/5fc1418c43774fc2bb9f494301d8430c 2023-04-21 01:53:59,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:59,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042099289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:59,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042099295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,306 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=708 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/cd2be2802e0848f4b0a80ee17485e5d1 2023-04-21 01:53:59,320 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=708 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/760d3e13a6074c6abc6f8222d0382a6a 2023-04-21 01:53:59,325 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/5fc1418c43774fc2bb9f494301d8430c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/5fc1418c43774fc2bb9f494301d8430c 2023-04-21 01:53:59,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/5fc1418c43774fc2bb9f494301d8430c, entries=250, sequenceid=708, filesize=16.7 K 2023-04-21 01:53:59,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/cd2be2802e0848f4b0a80ee17485e5d1 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/cd2be2802e0848f4b0a80ee17485e5d1 2023-04-21 01:53:59,335 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/cd2be2802e0848f4b0a80ee17485e5d1, entries=150, sequenceid=708, filesize=11.9 K 2023-04-21 01:53:59,337 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/760d3e13a6074c6abc6f8222d0382a6a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/760d3e13a6074c6abc6f8222d0382a6a 2023-04-21 01:53:59,341 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/760d3e13a6074c6abc6f8222d0382a6a, entries=150, sequenceid=708, filesize=11.9 K 2023-04-21 01:53:59,342 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=33.54 KB/34350 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 65ms, sequenceid=708, compaction requested=true 2023-04-21 01:53:59,342 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:59,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-21 01:53:59,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-21 01:53:59,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-21 01:53:59,352 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,352 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:59,352 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,352 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:59,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:59,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:59,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:59,355 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:59,355 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:59,355 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:59,355 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:59,355 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:59,390 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:59,390 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:59,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:59,390 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:59,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:59,391 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,392 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:59,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:59,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:53:59,398 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-21 01:53:59,399 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:53:59,399 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:59,399 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:53:59,399 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:59,399 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:53:59,399 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:53:59,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682042039425, max timestamp=1682042039442Num uniques 859; 2023-04-21 01:53:59,444 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:59,444 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:59,444 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,445 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:59,445 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:59,445 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,445 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:59,445 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:59,445 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:59,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:59,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042039429, max timestamp=1682042039442Num uniques 859; 2023-04-21 01:53:59,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:53:59,444 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:59,446 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:59,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:59,444 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:59,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682042039425, max timestamp=1682042039442Num uniques 859; 2023-04-21 01:53:59,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:59,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-21 01:53:59,446 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:59,446 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:59,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-21 01:53:59,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:59,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:59,486 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:59,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042099481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:59,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042099487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,547 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:53:59,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:53:59,549 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:53:59,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:53:59,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:53:59,556 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:53:59,556 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:59,556 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:53:59,556 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:53:59,557 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:59,557 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:59,588 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:59,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042099588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,592 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/cc4d1b2c5c3348ed900fce7e7ed285ef as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/cc4d1b2c5c3348ed900fce7e7ed285ef 2023-04-21 01:53:59,593 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/c4fda86e7b5e4572b63d4aab8c27d0e0 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/c4fda86e7b5e4572b63d4aab8c27d0e0 2023-04-21 01:53:59,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:59,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042099591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,600 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/B of 16b5c71a3c47da32fe9f26ec9e41dfb8 into c4fda86e7b5e4572b63d4aab8c27d0e0(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:59,600 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/A of 16b5c71a3c47da32fe9f26ec9e41dfb8 into cc4d1b2c5c3348ed900fce7e7ed285ef(size=13.0 K), total size for store is 29.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:59,600 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:59,600 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:59,600 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/B, priority=12, startTime=1682042039136; duration=0sec 2023-04-21 01:53:59,600 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/A, priority=13, startTime=1682042039136; duration=0sec 2023-04-21 01:53:59,600 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-21 01:53:59,600 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-21 01:53:59,600 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-21 01:53:59,602 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61850 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-21 01:53:59,602 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/C is initiating minor compaction (all files) 2023-04-21 01:53:59,602 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 5 compacting, 0 eligible, 16 blocking 2023-04-21 01:53:59,602 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/C in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:59,602 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-21 01:53:59,603 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-21 01:53:59,603 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. because compaction request was cancelled 2023-04-21 01:53:59,603 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-21 01:53:59,603 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/4fad11ba83554c0f908962810a493969, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/dd7e53da8a05472884294e290ce259ac, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/9a865824f87b4133a05cee5cdfca8921, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/6aa9e03a9a5b4e5c894ef500de98e268, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/760d3e13a6074c6abc6f8222d0382a6a] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=60.4 K 2023-04-21 01:53:59,604 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 4fad11ba83554c0f908962810a493969, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=624, earliestPutTs=1682042036682 2023-04-21 01:53:59,604 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-21 01:53:59,604 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-21 01:53:59,604 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. because compaction request was cancelled 2023-04-21 01:53:59,604 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-21 01:53:59,604 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting dd7e53da8a05472884294e290ce259ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1682042036740 2023-04-21 01:53:59,606 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-21 01:53:59,606 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-21 01:53:59,606 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. because compaction request was cancelled 2023-04-21 01:53:59,606 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 9a865824f87b4133a05cee5cdfca8921, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=660, earliestPutTs=1682042037420 2023-04-21 01:53:59,607 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 6aa9e03a9a5b4e5c894ef500de98e268, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1682042038543 2023-04-21 01:53:59,608 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 760d3e13a6074c6abc6f8222d0382a6a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=708, earliestPutTs=1682042038635 2023-04-21 01:53:59,621 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:59,621 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:59,621 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,622 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:53:59,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:53:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:59,622 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:53:59,625 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#C#compaction#101 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:53:59,627 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:53:59,627 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:53:59,627 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:53:59,627 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:53:59,627 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:53:59,628 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:53:59,628 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:53:59,628 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:59,628 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:53:59,628 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:53:59,628 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:53:59,629 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:53:59,629 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:53:59,629 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,629 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:53:59,629 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:53:59,647 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/8e33c09a45b942fba011cf231def2fa8 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/8e33c09a45b942fba011cf231def2fa8 2023-04-21 01:53:59,653 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/C of 16b5c71a3c47da32fe9f26ec9e41dfb8 into 8e33c09a45b942fba011cf231def2fa8(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:53:59,653 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:53:59,653 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/C, priority=11, startTime=1682042039342; duration=0sec 2023-04-21 01:53:59,654 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:53:59,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:59,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59306 deadline: 1682042099656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,658 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4169 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:53:59,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:59,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59272 deadline: 1682042099657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,661 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:53:59,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:59,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59276 deadline: 1682042099659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,663 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:53:59,706 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,706 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:59,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:59,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:59,706 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:59,706 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:53:59,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:59,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:59,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:59,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:59,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:53:59,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a26e2ed[Count = 0] remaining members to acquire global barrier 2023-04-21 01:53:59,708 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:53:59,708 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:59,774 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:59,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:59,775 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:53:59,775 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,775 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:53:59,775 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:53:59,775 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:53:59,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:59,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042099790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:53:59,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042099799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,815 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/b6ef1e9c01e04cc688259ed3db337e29 2023-04-21 01:53:59,827 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/f4884a0307c74f38821cd457d18eed6c 2023-04-21 01:53:59,845 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,845 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:53:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:59,845 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:59,845 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:53:59,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:59,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:59,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:59,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:59,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:59,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:53:59,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:53:59,848 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:53:59,849 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:53:59,849 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:53:59,849 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:53:59,849 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:53:59,921 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:59,921 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:59,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:53:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:53:59,921 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:53:59,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:53:59,922 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:53:59,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:59,923 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:53:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:53:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:59,967 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:53:59,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:53:59,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:53:59,967 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:53:59,967 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:53:59,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:53:59,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:59,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:53:59,968 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:53:59,968 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:53:59,968 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:53:59,968 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:53:59,968 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:53:59,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:53:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:53:59,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:53:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:00,068 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:00,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:00,070 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:00,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:00,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:00,072 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:00,072 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:00,072 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:00,072 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:00,072 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:00,073 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:00,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:00,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042100095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:00,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042100100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,149 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,149 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:00,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:00,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:00,150 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,150 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:00,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:00,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:00,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:00,151 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:00,151 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:00,151 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:00,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:00,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:00,152 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:00,152 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:54:00,152 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:54:00,152 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:00,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:00,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:00,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:00,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:00,190 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:00,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:00,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:00,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:00,191 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:00,191 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:00,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:00,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:00,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:00,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:00,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a29ffcd[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:00,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:00,194 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:00,246 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/c2923e7aca374fa18622bd11e0b55444 2023-04-21 01:54:00,252 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:00,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:00,252 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:00,252 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:00,252 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:00,252 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,252 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:00,253 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/b6ef1e9c01e04cc688259ed3db337e29 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b6ef1e9c01e04cc688259ed3db337e29 2023-04-21 01:54:00,259 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b6ef1e9c01e04cc688259ed3db337e29, entries=200, sequenceid=718, filesize=14.3 K 2023-04-21 01:54:00,260 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/f4884a0307c74f38821cd457d18eed6c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/f4884a0307c74f38821cd457d18eed6c 2023-04-21 01:54:00,266 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/f4884a0307c74f38821cd457d18eed6c, entries=100, sequenceid=718, filesize=9.5 K 2023-04-21 01:54:00,267 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/c2923e7aca374fa18622bd11e0b55444 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/c2923e7aca374fa18622bd11e0b55444 2023-04-21 01:54:00,274 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/c2923e7aca374fa18622bd11e0b55444, entries=100, sequenceid=718, filesize=9.5 K 2023-04-21 01:54:00,275 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 877ms, sequenceid=718, compaction requested=true 2023-04-21 01:54:00,275 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:54:00,275 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:00,275 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:00,275 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:00,275 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-21 01:54:00,276 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:00,277 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 44964 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:00,277 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/A is initiating minor compaction (all files) 2023-04-21 01:54:00,277 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/A in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:00,277 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/cc4d1b2c5c3348ed900fce7e7ed285ef, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/5fc1418c43774fc2bb9f494301d8430c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b6ef1e9c01e04cc688259ed3db337e29] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=43.9 K 2023-04-21 01:54:00,278 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting cc4d1b2c5c3348ed900fce7e7ed285ef, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1682042038543 2023-04-21 01:54:00,278 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35204 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:00,278 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/B is initiating minor compaction (all files) 2023-04-21 01:54:00,278 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/B in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:00,278 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/c4fda86e7b5e4572b63d4aab8c27d0e0, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/cd2be2802e0848f4b0a80ee17485e5d1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/f4884a0307c74f38821cd457d18eed6c] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=34.4 K 2023-04-21 01:54:00,279 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 5fc1418c43774fc2bb9f494301d8430c, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=708, earliestPutTs=1682042038635 2023-04-21 01:54:00,279 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting c4fda86e7b5e4572b63d4aab8c27d0e0, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1682042038543 2023-04-21 01:54:00,280 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting cd2be2802e0848f4b0a80ee17485e5d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=708, earliestPutTs=1682042038635 2023-04-21 01:54:00,280 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting b6ef1e9c01e04cc688259ed3db337e29, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1682042039287 2023-04-21 01:54:00,284 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting f4884a0307c74f38821cd457d18eed6c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1682042039287 2023-04-21 01:54:00,292 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#B#compaction#104 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:00,293 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#A#compaction#105 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:00,305 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,305 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:00,305 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:00,306 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:00,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:00,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:00,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:00,309 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:00,309 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:00,309 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:00,309 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:00,309 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:00,352 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:00,352 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:00,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:00,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:00,352 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:00,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:00,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:00,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:00,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:00,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:00,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:00,354 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,355 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:00,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:00,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,405 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,405 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:00,405 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:00,406 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:00,406 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,406 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:00,406 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:00,406 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:00,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:00,406 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:00,406 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:00,406 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:00,406 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:00,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:00,406 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:00,406 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:00,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:00,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:00,506 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:00,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:00,509 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:00,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:00,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:00,511 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:00,511 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:00,511 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:00,511 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:00,512 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:00,512 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:00,559 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,559 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:00,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:00,560 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,560 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:00,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:00,561 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:00,561 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:00,561 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:00,561 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:00,561 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:00,561 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:00,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:00,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:00,563 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:00,563 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:54:00,563 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-21 01:54:00,563 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:54:00,563 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:00,563 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:54:00,564 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:00,564 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:54:00,564 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:00,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:54:00,608 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:54:00,616 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=756 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/2958825b35704d6e86b71f55daf652d1 2023-04-21 01:54:00,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:00,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042100618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:00,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042100621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,634 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=756 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/d952a1a00b60435f88b533935df1abdb 2023-04-21 01:54:00,706 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/833099ab51cb45f084359b5cf9e69a99 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/833099ab51cb45f084359b5cf9e69a99 2023-04-21 01:54:00,713 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/e77f1ea4d6d149639016b19052450f93 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/e77f1ea4d6d149639016b19052450f93 2023-04-21 01:54:00,715 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/B of 16b5c71a3c47da32fe9f26ec9e41dfb8 into 833099ab51cb45f084359b5cf9e69a99(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:00,715 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:54:00,715 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/B, priority=13, startTime=1682042040275; duration=0sec 2023-04-21 01:54:00,716 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:00,716 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-21 01:54:00,717 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-21 01:54:00,717 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-21 01:54:00,718 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. because compaction request was cancelled 2023-04-21 01:54:00,720 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/A of 16b5c71a3c47da32fe9f26ec9e41dfb8 into e77f1ea4d6d149639016b19052450f93(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:00,720 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:54:00,721 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/A, priority=13, startTime=1682042040275; duration=0sec 2023-04-21 01:54:00,721 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:00,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:00,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042100722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:00,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042100725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:00,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042100927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:00,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:00,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042100927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,049 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=756 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/f09e3a15bed6458aa94c1532c511a18f 2023-04-21 01:54:01,054 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/2958825b35704d6e86b71f55daf652d1 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/2958825b35704d6e86b71f55daf652d1 2023-04-21 01:54:01,059 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/2958825b35704d6e86b71f55daf652d1, entries=150, sequenceid=756, filesize=11.9 K 2023-04-21 01:54:01,060 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/d952a1a00b60435f88b533935df1abdb as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/d952a1a00b60435f88b533935df1abdb 2023-04-21 01:54:01,065 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/d952a1a00b60435f88b533935df1abdb, entries=150, sequenceid=756, filesize=11.9 K 2023-04-21 01:54:01,066 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/f09e3a15bed6458aa94c1532c511a18f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/f09e3a15bed6458aa94c1532c511a18f 2023-04-21 01:54:01,077 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/f09e3a15bed6458aa94c1532c511a18f, entries=150, sequenceid=756, filesize=11.9 K 2023-04-21 01:54:01,078 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 515ms, sequenceid=756, compaction requested=true 2023-04-21 01:54:01,078 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:54:01,078 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:01,078 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:01,078 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:01,078 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,084 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:01,084 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:01,144 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,144 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:01,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:01,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:01,144 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:01,144 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:01,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:01,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:01,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:01,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:01,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:01,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d3178f0[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:01,146 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:01,146 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:01,198 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:01,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:01,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:01,198 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:01,198 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:01,198 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:01,198 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,198 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:01,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:54:01,234 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-21 01:54:01,235 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:54:01,235 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:01,235 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:54:01,235 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:01,235 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:54:01,235 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:01,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:01,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042101301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:01,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042101301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,390 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:01,390 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,390 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:01,390 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:01,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:01,394 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:01,394 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:01,394 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:01,394 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:01,394 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:01,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:01,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042101403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:01,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042101406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,437 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:01,437 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:01,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:01,437 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:01,437 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:01,439 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:01,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:01,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,490 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,490 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:01,490 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:01,490 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:01,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:01,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:01,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:01,490 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:01,491 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:01,490 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:01,490 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:01,491 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,491 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:01,491 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:01,491 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:01,491 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:01,491 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:01,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:01,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:01,591 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:01,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:01,593 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:01,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:01,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:01,595 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:01,595 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:01,595 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:01,595 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:01,596 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:01,596 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:01,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:01,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042101605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:01,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042101608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,659 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,659 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:01,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:01,660 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,660 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:01,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:01,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:01,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:01,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:01,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:01,661 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:01,661 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:01,661 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:01,661 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:01,661 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:01,661 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:01,661 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:54:01,661 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:54:01,661 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:01,662 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:01,662 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:01,662 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,662 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:01,662 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:01,721 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:01,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:01,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:01,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:01,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:01,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:01,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:01,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eb5ac57[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:01,723 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:01,723 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:01,724 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:01,725 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=770 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/2f62574fceaa4234bec2dc8c2a830b90 2023-04-21 01:54:01,790 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:01,791 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:01,791 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:01,791 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:01,791 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:01,791 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,791 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:01,791 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:01,852 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:01,852 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,852 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:01,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:01,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:01,852 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:01,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:01,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:01,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:01,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:01,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:01,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:01,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:01,854 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:01,854 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:01,854 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:01,854 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:01,854 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:01,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:01,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042101908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:01,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042101913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,936 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:01,936 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:01,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:01,936 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:01,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:01,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:01,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:01,937 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:01,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:01,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:01,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:01,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,990 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:01,990 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:01,990 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,990 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:01,990 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:01,990 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:01,990 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:01,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:01,990 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:01,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:01,991 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:01,991 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:01,991 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:01,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:01,991 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:01,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:01,991 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:01,991 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:01,991 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:01,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:02,091 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:02,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:02,094 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:02,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:02,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:02,095 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:02,095 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:02,096 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:02,096 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:02,096 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,096 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:02,143 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=770 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/f62de8a1a68d43038645a8aa16e1b039 2023-04-21 01:54:02,144 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:02,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:02,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:02,144 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:02,145 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,145 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:02,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:02,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:02,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:02,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:02,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:02,147 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:02,147 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:02,147 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:02,154 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:02,157 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:02,157 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:02,157 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:54:02,157 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:54:02,157 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:02,157 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:02,157 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:02,157 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,157 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:02,157 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:02,165 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=770 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/28e0e530c8624ac8b4e92d021226f2d7 2023-04-21 01:54:02,169 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/2f62574fceaa4234bec2dc8c2a830b90 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/2f62574fceaa4234bec2dc8c2a830b90 2023-04-21 01:54:02,175 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/2f62574fceaa4234bec2dc8c2a830b90, entries=200, sequenceid=770, filesize=14.3 K 2023-04-21 01:54:02,176 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/f62de8a1a68d43038645a8aa16e1b039 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/f62de8a1a68d43038645a8aa16e1b039 2023-04-21 01:54:02,180 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/f62de8a1a68d43038645a8aa16e1b039, entries=100, sequenceid=770, filesize=9.5 K 2023-04-21 01:54:02,180 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/28e0e530c8624ac8b4e92d021226f2d7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/28e0e530c8624ac8b4e92d021226f2d7 2023-04-21 01:54:02,186 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/28e0e530c8624ac8b4e92d021226f2d7, entries=100, sequenceid=770, filesize=9.5 K 2023-04-21 01:54:02,187 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 953ms, sequenceid=770, compaction requested=true 2023-04-21 01:54:02,187 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:54:02,187 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:02,187 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:02,187 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:02,187 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:02,187 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:02,188 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35306 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:02,188 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/B is initiating minor compaction (all files) 2023-04-21 01:54:02,188 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40186 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:02,188 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/B in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:02,188 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/A is initiating minor compaction (all files) 2023-04-21 01:54:02,189 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/A in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:02,189 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/833099ab51cb45f084359b5cf9e69a99, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/d952a1a00b60435f88b533935df1abdb, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/f62de8a1a68d43038645a8aa16e1b039] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=34.5 K 2023-04-21 01:54:02,189 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/e77f1ea4d6d149639016b19052450f93, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/2958825b35704d6e86b71f55daf652d1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/2f62574fceaa4234bec2dc8c2a830b90] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=39.2 K 2023-04-21 01:54:02,189 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting e77f1ea4d6d149639016b19052450f93, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1682042038638 2023-04-21 01:54:02,189 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 833099ab51cb45f084359b5cf9e69a99, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1682042038638 2023-04-21 01:54:02,190 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 2958825b35704d6e86b71f55daf652d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=756, earliestPutTs=1682042039459 2023-04-21 01:54:02,190 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting d952a1a00b60435f88b533935df1abdb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=756, earliestPutTs=1682042039459 2023-04-21 01:54:02,190 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 2f62574fceaa4234bec2dc8c2a830b90, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1682042040617 2023-04-21 01:54:02,190 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting f62de8a1a68d43038645a8aa16e1b039, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1682042040617 2023-04-21 01:54:02,209 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#B#compaction#112 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:02,210 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#A#compaction#113 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:02,221 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,221 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:02,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:02,222 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,222 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:02,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:02,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:02,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:02,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:02,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@736914e6[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:02,223 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:02,223 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,244 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,244 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:02,244 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:02,244 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:02,245 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,245 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:02,249 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/6bca07c3085b4be48b0c5bf2bed2a858 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/6bca07c3085b4be48b0c5bf2bed2a858 2023-04-21 01:54:02,256 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/A of 16b5c71a3c47da32fe9f26ec9e41dfb8 into 6bca07c3085b4be48b0c5bf2bed2a858(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:02,256 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:54:02,256 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/A, priority=13, startTime=1682042042187; duration=0sec 2023-04-21 01:54:02,256 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:02,256 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:54:02,259 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 44946 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:54:02,259 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/C is initiating minor compaction (all files) 2023-04-21 01:54:02,259 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/C in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:02,259 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/8e33c09a45b942fba011cf231def2fa8, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/c2923e7aca374fa18622bd11e0b55444, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/f09e3a15bed6458aa94c1532c511a18f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/28e0e530c8624ac8b4e92d021226f2d7] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=43.9 K 2023-04-21 01:54:02,259 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 8e33c09a45b942fba011cf231def2fa8, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=708, earliestPutTs=1682042038635 2023-04-21 01:54:02,260 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting c2923e7aca374fa18622bd11e0b55444, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1682042039287 2023-04-21 01:54:02,261 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting f09e3a15bed6458aa94c1532c511a18f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=756, earliestPutTs=1682042039459 2023-04-21 01:54:02,261 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 28e0e530c8624ac8b4e92d021226f2d7, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1682042040617 2023-04-21 01:54:02,273 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#C#compaction#114 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:02,299 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:02,299 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:02,299 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:02,299 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:02,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:02,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:02,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:02,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:02,306 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:02,306 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:02,306 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:02,306 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:02,306 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:02,310 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/8c5b991cb3bb4a9da9f4d2f1bbc7785d as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/8c5b991cb3bb4a9da9f4d2f1bbc7785d 2023-04-21 01:54:02,317 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/C of 16b5c71a3c47da32fe9f26ec9e41dfb8 into 8c5b991cb3bb4a9da9f4d2f1bbc7785d(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:02,317 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:54:02,317 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/C, priority=12, startTime=1682042042187; duration=0sec 2023-04-21 01:54:02,317 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:02,367 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,367 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:02,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:02,367 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:02,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:02,368 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:02,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:02,372 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:02,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:02,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:02,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,399 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,399 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:02,400 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:02,400 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:02,400 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:02,400 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:02,400 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:02,400 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:02,400 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:02,400 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:02,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:02,400 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,401 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,401 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,401 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:02,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:02,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:02,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:54:02,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042041297, max timestamp=1682042042411Num uniques 859; 2023-04-21 01:54:02,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042041297, max timestamp=1682042042411Num uniques 859; 2023-04-21 01:54:02,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042041297, max timestamp=1682042042411Num uniques 859; 2023-04-21 01:54:02,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:54:02,413 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=58.14 KB heapSize=164.17 KB 2023-04-21 01:54:02,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:54:02,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:54:02,414 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:54:02,414 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:02,414 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:54:02,414 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:02,414 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:54:02,414 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:02,446 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=793 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/5022dc45ced2448a9d8dbd791bcfc4d5 2023-04-21 01:54:02,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682042042446, max timestamp=1682042042454Num uniques 859; 2023-04-21 01:54:02,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:54:02,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682042042446, max timestamp=1682042042455Num uniques 859; 2023-04-21 01:54:02,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:54:02,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682042042446, max timestamp=1682042042455Num uniques 859; 2023-04-21 01:54:02,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:54:02,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682042042476, max timestamp=1682042042489Num uniques 1068; 2023-04-21 01:54:02,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682042042476, max timestamp=1682042042489Num uniques 1068; 2023-04-21 01:54:02,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-21 01:54:02,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-21 01:54:02,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682042042476, max timestamp=1682042042489Num uniques 1068; 2023-04-21 01:54:02,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-21 01:54:02,500 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:02,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:02,503 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:02,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:02,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:02,505 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:02,505 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:02,505 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:02,505 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:02,505 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,505 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:02,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1682042042500, max timestamp=1682042042523Num uniques 1027; 2023-04-21 01:54:02,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=17262, suffix heap size=134776,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=377 2023-04-21 01:54:02,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1682042042500, max timestamp=1682042042523Num uniques 1027; 2023-04-21 01:54:02,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=17262, suffix heap size=134776,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=377 2023-04-21 01:54:02,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1682042042500, max timestamp=1682042042523Num uniques 1027; 2023-04-21 01:54:02,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=17262, suffix heap size=134776,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=377 2023-04-21 01:54:02,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8506, totalHeapSize=19584, min timestamp=1682042042549, max timestamp=1682042042560Num uniques 1236; 2023-04-21 01:54:02,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=56596, new segment data size=8506, suffix heap size=156512,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=186 2023-04-21 01:54:02,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8511, totalHeapSize=19584, min timestamp=1682042042549, max timestamp=1682042042560Num uniques 1236; 2023-04-21 01:54:02,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=56601, new segment data size=8511, suffix heap size=156512,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=186 2023-04-21 01:54:02,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8506, totalHeapSize=19584, min timestamp=1682042042549, max timestamp=1682042042560Num uniques 1236; 2023-04-21 01:54:02,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=56596, new segment data size=8506, suffix heap size=156512,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=186 2023-04-21 01:54:02,567 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:02,567 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:02,568 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:02,568 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:02,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:02,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:02,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:02,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:02,569 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:02,569 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:02,569 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:02,569 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:02,569 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:02,570 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:02,570 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:54:02,570 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:54:02,570 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:02,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682042042582, max timestamp=1682042042588Num uniques 1045; 2023-04-21 01:54:02,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682042042582, max timestamp=1682042042588Num uniques 1045; 2023-04-21 01:54:02,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-21 01:54:02,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-21 01:54:02,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682042042582, max timestamp=1682042042588Num uniques 1045; 2023-04-21 01:54:02,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-21 01:54:02,613 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,613 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:02,614 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,614 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:02,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:02,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cb4ad1[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:02,616 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:02,616 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682042042612, max timestamp=1682042042620Num uniques 1104; 2023-04-21 01:54:02,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682042042612, max timestamp=1682042042620Num uniques 1104; 2023-04-21 01:54:02,624 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-21 01:54:02,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682042042612, max timestamp=1682042042620Num uniques 1104; 2023-04-21 01:54:02,624 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-21 01:54:02,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-21 01:54:02,652 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/9410f60fc7244e689a4f385bf03dc7e7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/9410f60fc7244e689a4f385bf03dc7e7 2023-04-21 01:54:02,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682042042644, max timestamp=1682042042657Num uniques 1013; 2023-04-21 01:54:02,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-21 01:54:02,660 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/B of 16b5c71a3c47da32fe9f26ec9e41dfb8 into 9410f60fc7244e689a4f385bf03dc7e7(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:02,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682042042644, max timestamp=1682042042657Num uniques 1013; 2023-04-21 01:54:02,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682042042644, max timestamp=1682042042657Num uniques 1013; 2023-04-21 01:54:02,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-21 01:54:02,660 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:54:02,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-21 01:54:02,660 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/B, priority=13, startTime=1682042042187; duration=0sec 2023-04-21 01:54:02,661 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:02,698 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,698 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:02,698 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:02,698 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,698 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:02,698 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:02,767 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,767 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:02,767 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:02,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:02,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:02,767 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:02,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:02,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:02,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:02,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:02,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:02,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:02,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:02,770 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:02,771 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:02,771 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:02,771 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:02,771 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:02,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682042042679, max timestamp=1682042042770Num uniques 1022; 2023-04-21 01:54:02,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682042042679, max timestamp=1682042042770Num uniques 1022; 2023-04-21 01:54:02,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-21 01:54:02,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-21 01:54:02,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682042042679, max timestamp=1682042042770Num uniques 1022; 2023-04-21 01:54:02,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-21 01:54:02,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=381, cellSize=17446, totalHeapSize=39864, min timestamp=1682042042786, max timestamp=1682042042793Num uniques 1081; 2023-04-21 01:54:02,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=381, cellSize=17446, totalHeapSize=39864, min timestamp=1682042042786, max timestamp=1682042042793Num uniques 1031; 2023-04-21 01:54:02,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=17446, suffix heap size=140392,new segment heap size=39864 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=381 2023-04-21 01:54:02,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=17446, suffix heap size=135192,new segment heap size=39864 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=381 2023-04-21 01:54:02,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=381, cellSize=17446, totalHeapSize=39864, min timestamp=1682042042786, max timestamp=1682042042793Num uniques 1081; 2023-04-21 01:54:02,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=17446, suffix heap size=140392,new segment heap size=39864 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=381 2023-04-21 01:54:02,805 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,806 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:02,806 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:02,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:02,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:02,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:02,806 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:02,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,808 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:02,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1682042042804, max timestamp=1682042042818Num uniques 1240; 2023-04-21 01:54:02,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1682042042804, max timestamp=1682042042818Num uniques 1240; 2023-04-21 01:54:02,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=56790, new segment data size=13280, suffix heap size=156928,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1240, new segment cells count=290 2023-04-21 01:54:02,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1682042042804, max timestamp=1682042042818Num uniques 1240; 2023-04-21 01:54:02,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=56790, new segment data size=13280, suffix heap size=156928,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1240, new segment cells count=290 2023-04-21 01:54:02,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=56790, new segment data size=13280, suffix heap size=156928,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1240, new segment cells count=290 2023-04-21 01:54:02,831 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,831 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:02,831 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:02,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:02,831 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:02,831 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:02,832 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:02,831 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:02,832 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,832 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,832 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,832 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:02,832 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:02,832 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:02,832 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:02,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:02,874 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=793 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/a96e214169a24170b436eb41eddb503e 2023-04-21 01:54:02,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682042042847, max timestamp=1682042042855Num uniques 1149; 2023-04-21 01:54:02,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=9114, suffix heap size=147464,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=199 2023-04-21 01:54:02,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682042042847, max timestamp=1682042042855Num uniques 1149; 2023-04-21 01:54:02,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=9114, suffix heap size=147464,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=199 2023-04-21 01:54:02,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682042042847, max timestamp=1682042042855Num uniques 1149; 2023-04-21 01:54:02,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=9114, suffix heap size=147464,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=199 2023-04-21 01:54:02,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682042042888, max timestamp=1682042042901Num uniques 1058; 2023-04-21 01:54:02,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-21 01:54:02,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682042042888, max timestamp=1682042042901Num uniques 1058; 2023-04-21 01:54:02,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682042042888, max timestamp=1682042042901Num uniques 1058; 2023-04-21 01:54:02,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-21 01:54:02,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-21 01:54:02,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682042042909, max timestamp=1682042042927Num uniques 1067; 2023-04-21 01:54:02,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-21 01:54:02,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682042042909, max timestamp=1682042042927Num uniques 1067; 2023-04-21 01:54:02,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-21 01:54:02,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682042042909, max timestamp=1682042042927Num uniques 1067; 2023-04-21 01:54:02,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-21 01:54:02,932 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:02,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:02,934 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:02,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:02,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:02,936 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:02,936 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:02,936 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:02,936 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:02,937 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,937 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:02,938 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:02,938 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:02,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:02,938 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,938 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:02,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:02,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:02,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:02,939 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:02,939 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:02,940 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:02,940 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:02,940 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:02,941 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:02,941 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:02,941 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:02,941 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:54:02,941 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:54:02,941 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:02,941 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:02,941 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:02,941 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,941 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:02,941 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:02,944 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,944 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:02,944 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:02,944 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:02,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:02,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@201b5400[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:02,946 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:02,946 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,948 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,948 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:02,948 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:02,948 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:02,948 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,948 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:02,950 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,950 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:02,950 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:02,950 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:02,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:02,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:02,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:02,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:02,954 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:02,954 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:02,954 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:02,954 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:02,954 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:02,956 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,956 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:02,956 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:02,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,957 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:02,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:02,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:02,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:02,960 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:02,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:02,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,968 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:02,968 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:02,968 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:02,968 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:02,968 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:02,968 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:02,968 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:02,968 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,968 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:02,968 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:02,968 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:02,968 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:02,968 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:02,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:02,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:02,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:02,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:02,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:02,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682042042960, max timestamp=1682042042970Num uniques 1076; 2023-04-21 01:54:02,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682042042960, max timestamp=1682042042970Num uniques 1076; 2023-04-21 01:54:02,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-21 01:54:02,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682042042960, max timestamp=1682042042970Num uniques 1076; 2023-04-21 01:54:02,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-21 01:54:02,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-21 01:54:03,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1682042042994, max timestamp=1682042043001Num uniques 1085; 2023-04-21 01:54:03,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=8470, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-04-21 01:54:03,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1682042042994, max timestamp=1682042043001Num uniques 1085; 2023-04-21 01:54:03,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=8470, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-04-21 01:54:03,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1682042042994, max timestamp=1682042043001Num uniques 1085; 2023-04-21 01:54:03,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=8470, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-04-21 01:54:03,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1682042043020, max timestamp=1682042043030Num uniques 1044; 2023-04-21 01:54:03,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1682042043020, max timestamp=1682042043030Num uniques 1044; 2023-04-21 01:54:03,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1682042043020, max timestamp=1682042043030Num uniques 1044; 2023-04-21 01:54:03,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-21 01:54:03,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-21 01:54:03,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-21 01:54:03,068 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:03,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:03,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1682042043058, max timestamp=1682042043066Num uniques 1103; 2023-04-21 01:54:03,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=7005, suffix heap size=142680,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=153 2023-04-21 01:54:03,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1682042043058, max timestamp=1682042043067Num uniques 1103; 2023-04-21 01:54:03,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=7005, suffix heap size=142680,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=153 2023-04-21 01:54:03,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1682042043058, max timestamp=1682042043066Num uniques 1103; 2023-04-21 01:54:03,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=7005, suffix heap size=142680,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=153 2023-04-21 01:54:03,073 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:03,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:03,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:03,075 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:03,075 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:03,075 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:03,075 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:03,075 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,075 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,077 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:03,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:03,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:03,078 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,078 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,079 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:03,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:03,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,079 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:03,079 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:03,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:03,079 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:03,079 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:03,080 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:03,092 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:03,092 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:03,092 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:03,092 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:54:03,092 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:54:03,092 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:03,092 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:03,092 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:03,092 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,092 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:03,093 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:03,095 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,095 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:03,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:03,096 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,096 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:03,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:03,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:03,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1682042043087, max timestamp=1682042043097Num uniques 1012; 2023-04-21 01:54:03,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1682042043087, max timestamp=1682042043097Num uniques 1012; 2023-04-21 01:54:03,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=11992, suffix heap size=133216,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=262 2023-04-21 01:54:03,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1682042043087, max timestamp=1682042043097Num uniques 1012; 2023-04-21 01:54:03,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=11992, suffix heap size=133216,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=262 2023-04-21 01:54:03,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=11992, suffix heap size=133216,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=262 2023-04-21 01:54:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:03,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:03,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@137f8340[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:03,101 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:03,101 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,103 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,103 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:03,103 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:03,103 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:03,103 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,103 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:03,104 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:03,105 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:03,105 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:03,105 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:03,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:03,109 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:03,109 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:03,109 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:03,109 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:03,109 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:03,111 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,111 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:03,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:03,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,111 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:03,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:03,111 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:03,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:03,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:03,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,112 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,118 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,118 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,118 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,118 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,119 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,119 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,119 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,119 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:03,119 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:03,119 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:03,119 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:03,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:03,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:03,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:03,119 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:03,119 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:03,119 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:03,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:03,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:03,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1682042043124, max timestamp=1682042043131Num uniques 1121; 2023-04-21 01:54:03,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1682042043124, max timestamp=1682042043131Num uniques 1121; 2023-04-21 01:54:03,133 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:03,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1682042043124, max timestamp=1682042043131Num uniques 1121; 2023-04-21 01:54:03,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=7826, suffix heap size=144552,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=171 2023-04-21 01:54:03,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=7826, suffix heap size=144552,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=171 2023-04-21 01:54:03,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=7826, suffix heap size=144552,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=171 2023-04-21 01:54:03,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042103132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,175 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1682042043153, max timestamp=1682042043173Num uniques 1030; 2023-04-21 01:54:03,175 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=10530, suffix heap size=135088,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=230 2023-04-21 01:54:03,175 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1682042043153, max timestamp=1682042043173Num uniques 1030; 2023-04-21 01:54:03,175 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1682042043153, max timestamp=1682042043173Num uniques 1030; 2023-04-21 01:54:03,175 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=10530, suffix heap size=135088,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=230 2023-04-21 01:54:03,175 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=10530, suffix heap size=135088,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=230 2023-04-21 01:54:03,219 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:03,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:03,222 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:03,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:03,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:03,223 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:03,223 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:03,223 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:03,223 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:03,224 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,224 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,225 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,225 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:03,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:03,225 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,225 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:03,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:03,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:03,227 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:03,227 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:03,227 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:03,227 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:54:03,227 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:54:03,228 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:03,228 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:03,228 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:03,228 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,228 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:03,228 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:03,229 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:03,230 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:03,230 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,230 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:03,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:03,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@280a6bed[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:03,231 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:03,231 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1682042043203, max timestamp=1682042043232Num uniques 1089; 2023-04-21 01:54:03,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=10944, suffix heap size=141224,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=239 2023-04-21 01:54:03,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1682042043203, max timestamp=1682042043232Num uniques 1089; 2023-04-21 01:54:03,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1682042043203, max timestamp=1682042043232Num uniques 1089; 2023-04-21 01:54:03,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=10944, suffix heap size=141224,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=239 2023-04-21 01:54:03,234 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=10944, suffix heap size=141224,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=239 2023-04-21 01:54:03,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,235 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:03,235 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:03,235 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:03,235 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,235 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:03,238 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,238 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:03,238 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:03,238 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:03,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:03,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:03,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:03,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:03,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:03,241 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:03,241 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:03,241 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:03,241 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:03,241 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:03,243 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,243 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:03,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,243 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:03,244 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:03,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:03,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:03,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,245 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,250 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:03,250 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:03,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:03,250 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:03,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:03,250 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:03,250 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:03,250 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:03,250 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:03,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:03,250 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,251 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,251 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,251 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,251 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,251 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,251 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1682042043245, max timestamp=1682042043257Num uniques 1098; 2023-04-21 01:54:03,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-04-21 01:54:03,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1682042043245, max timestamp=1682042043257Num uniques 1098; 2023-04-21 01:54:03,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-04-21 01:54:03,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1682042043245, max timestamp=1682042043257Num uniques 1098; 2023-04-21 01:54:03,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-04-21 01:54:03,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1682042043275, max timestamp=1682042043286Num uniques 1107; 2023-04-21 01:54:03,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=9475, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-04-21 01:54:03,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1682042043275, max timestamp=1682042043286Num uniques 1107; 2023-04-21 01:54:03,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=9475, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-04-21 01:54:03,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1682042043275, max timestamp=1682042043286Num uniques 1107; 2023-04-21 01:54:03,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=9475, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-04-21 01:54:03,290 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=793 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/85f45fcf88af492981c753c8cde8ec08 2023-04-21 01:54:03,296 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/5022dc45ced2448a9d8dbd791bcfc4d5 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/5022dc45ced2448a9d8dbd791bcfc4d5 2023-04-21 01:54:03,300 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/5022dc45ced2448a9d8dbd791bcfc4d5, entries=200, sequenceid=793, filesize=14.3 K 2023-04-21 01:54:03,301 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/a96e214169a24170b436eb41eddb503e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/a96e214169a24170b436eb41eddb503e 2023-04-21 01:54:03,306 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/a96e214169a24170b436eb41eddb503e, entries=150, sequenceid=793, filesize=11.9 K 2023-04-21 01:54:03,307 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/85f45fcf88af492981c753c8cde8ec08 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/85f45fcf88af492981c753c8cde8ec08 2023-04-21 01:54:03,312 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/85f45fcf88af492981c753c8cde8ec08, entries=150, sequenceid=793, filesize=11.9 K 2023-04-21 01:54:03,313 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.19 KB/67776, currentSize=127.47 KB/130530 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 900ms, sequenceid=793, compaction requested=false 2023-04-21 01:54:03,313 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:54:03,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:54:03,313 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=140.89 KB heapSize=400.41 KB 2023-04-21 01:54:03,314 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:54:03,314 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:03,314 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:54:03,314 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:03,314 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:54:03,314 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:03,322 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1209 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/96e871f9f9204bbe89a3cd8eaf4c3a16 2023-04-21 01:54:03,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:03,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 548 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042103326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:03,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 379 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042103326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,351 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:03,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:03,353 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:03,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:03,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:03,354 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:03,354 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:03,354 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:03,354 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:03,355 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,355 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,357 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,357 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:03,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:03,357 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,358 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:03,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:03,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:03,359 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:03,359 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:03,359 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:03,359 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:03,359 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:03,359 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:03,359 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:54:03,359 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:54:03,359 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:03,359 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:03,359 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:03,360 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,360 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:03,360 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:03,361 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,361 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:03,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:03,362 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,362 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:03,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:03,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@744b21ff[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:03,363 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:03,363 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,365 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,365 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:03,365 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:03,365 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:03,365 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,365 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:03,366 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,366 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:03,366 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:03,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:03,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:03,367 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:03,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:03,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:03,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:03,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:03,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:03,371 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:03,371 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:03,371 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:03,371 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:03,371 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:03,373 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,373 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:03,373 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:03,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:03,374 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:03,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,375 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:03,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,382 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,382 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,382 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:03,382 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:03,382 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:03,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:03,382 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,383 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:03,382 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:03,383 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,383 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,383 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,383 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,383 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:03,383 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:03,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:03,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:03,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:03,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042103431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:03,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 550 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042103432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,483 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:03,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:03,486 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:03,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:03,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:03,488 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:03,488 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:03,488 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:03,488 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:03,488 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,488 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,490 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:03,490 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:03,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:03,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,490 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,490 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:03,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:03,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:03,492 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:03,492 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:03,492 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:54:03,492 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:54:03,492 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:03,493 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:03,494 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,494 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:03,495 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,495 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:03,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:03,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:03,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:03,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e8ac022[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:03,497 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:03,497 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,499 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,499 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:03,499 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,499 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:03,499 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:03,499 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:03,502 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,502 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:03,502 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:03,502 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:03,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:03,505 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:03,505 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:03,505 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:03,506 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:03,506 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:03,507 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,507 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:03,507 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:03,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:03,508 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,509 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:03,511 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:03,511 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,511 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,511 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:03,511 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:03,511 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:03,511 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:03,511 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,512 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,512 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:03,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:03,511 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:03,512 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:03,512 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,512 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:03,612 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:03,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:03,614 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:03,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:03,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:03,616 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:03,616 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:03,616 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:03,616 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:03,617 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,617 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:03,619 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:03,619 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,619 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:03,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,620 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:03,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:03,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:03,620 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:03,620 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:03,620 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:03,621 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:03,621 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:03,621 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:03,621 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:54:03,621 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:54:03,621 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:03,621 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:03,621 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:03,621 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,621 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:03,621 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:03,623 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,623 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:03,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:03,623 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,623 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:03,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:03,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:03,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61002730[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:03,625 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:03,625 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,626 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,627 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:03,627 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:03,627 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:03,627 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,627 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:03,628 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:03,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:03,628 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:03,628 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:03,628 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:03,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:03,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:03,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:03,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:03,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:03,631 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:03,631 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:03,631 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:03,631 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:03,631 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:03,633 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,633 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:03,633 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:03,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:03,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,634 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:03,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:03,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:03,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:03,635 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:03,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042103634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,637 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,637 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,637 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,637 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,637 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:03,637 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,637 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:03,638 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:03,637 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:03,638 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:03,638 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,638 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:03,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:03,638 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:03,638 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:03,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:03,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:03,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:03,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:03,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 552 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042103639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:03,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59272 deadline: 1682042103668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,673 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8184 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:54:03,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:03,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59306 deadline: 1682042103679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,682 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8193 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:54:03,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:03,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59276 deadline: 1682042103689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,692 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8201 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:54:03,735 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1209 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/b2f082f8c24441daa681eb91eb168b35 2023-04-21 01:54:03,738 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:03,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:03,741 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:03,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:03,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:03,743 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:03,743 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:03,743 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:03,743 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:03,743 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,743 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,747 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:03,747 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:03,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:03,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,752 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,752 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:03,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:03,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:03,753 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:03,753 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:03,753 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:03,753 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:03,753 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:03,753 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:03,753 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:03,753 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:03,753 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:54:03,753 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:54:03,753 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:03,753 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:03,753 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:03,754 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,754 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:03,754 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:03,755 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1209 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/32a9121b838f4df19dcfc4406961a7eb 2023-04-21 01:54:03,756 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,756 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:03,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:03,757 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,757 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:03,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:03,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:03,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:03,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:03,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aae6dfe[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:03,758 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:03,759 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,760 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,760 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:03,760 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:03,760 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:03,760 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/96e871f9f9204bbe89a3cd8eaf4c3a16 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/96e871f9f9204bbe89a3cd8eaf4c3a16 2023-04-21 01:54:03,761 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,761 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:03,762 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,762 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:03,762 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:03,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:03,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:03,762 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:03,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:03,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:03,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:03,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:03,765 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:03,765 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:03,765 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:03,765 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:03,765 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:03,766 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,766 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:03,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:03,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,767 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:03,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:03,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:03,767 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:03,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:03,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,768 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:03,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:03,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,771 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,771 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:03,771 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,771 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:03,771 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,771 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:03,771 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:03,771 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:03,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:03,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:03,771 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,772 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,772 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:03,772 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,772 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:03,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:03,772 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:03,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:03,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:03,778 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/96e871f9f9204bbe89a3cd8eaf4c3a16, entries=300, sequenceid=1209, filesize=19.0 K 2023-04-21 01:54:03,779 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/b2f082f8c24441daa681eb91eb168b35 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/b2f082f8c24441daa681eb91eb168b35 2023-04-21 01:54:03,784 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/b2f082f8c24441daa681eb91eb168b35, entries=150, sequenceid=1209, filesize=11.9 K 2023-04-21 01:54:03,785 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/32a9121b838f4df19dcfc4406961a7eb as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/32a9121b838f4df19dcfc4406961a7eb 2023-04-21 01:54:03,790 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/32a9121b838f4df19dcfc4406961a7eb, entries=150, sequenceid=1209, filesize=11.9 K 2023-04-21 01:54:03,791 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.36 KB/409968, currentSize=40.25 KB/41220 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 478ms, sequenceid=1209, compaction requested=true 2023-04-21 01:54:03,792 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:54:03,792 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:03,792 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:03,792 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:03,792 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:03,792 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:03,794 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 47612 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:03,794 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/A is initiating minor compaction (all files) 2023-04-21 01:54:03,794 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/A in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:03,794 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/6bca07c3085b4be48b0c5bf2bed2a858, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/5022dc45ced2448a9d8dbd791bcfc4d5, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/96e871f9f9204bbe89a3cd8eaf4c3a16] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=46.5 K 2023-04-21 01:54:03,795 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:03,795 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/B is initiating minor compaction (all files) 2023-04-21 01:54:03,795 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/B in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:03,795 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/9410f60fc7244e689a4f385bf03dc7e7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/a96e214169a24170b436eb41eddb503e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/b2f082f8c24441daa681eb91eb168b35] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=37.0 K 2023-04-21 01:54:03,798 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 9410f60fc7244e689a4f385bf03dc7e7, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1682042039474 2023-04-21 01:54:03,798 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 6bca07c3085b4be48b0c5bf2bed2a858, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1682042039474 2023-04-21 01:54:03,798 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting a96e214169a24170b436eb41eddb503e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=793, earliestPutTs=1682042041297 2023-04-21 01:54:03,798 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 5022dc45ced2448a9d8dbd791bcfc4d5, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=793, earliestPutTs=1682042041297 2023-04-21 01:54:03,799 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting b2f082f8c24441daa681eb91eb168b35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1209, earliestPutTs=1682042043301 2023-04-21 01:54:03,800 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 96e871f9f9204bbe89a3cd8eaf4c3a16, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=1209, earliestPutTs=1682042043301 2023-04-21 01:54:03,812 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#A#compaction#121 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:03,812 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#B#compaction#122 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:03,872 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:03,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:03,874 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:03,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:03,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:03,875 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:03,875 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:03,876 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:03,876 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:03,876 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,876 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,877 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,877 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:03,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:03,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:03,878 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,878 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:03,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:03,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:03,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:03,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:03,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:03,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:03,879 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:03,879 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:03,879 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:03,879 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:03,879 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:03,879 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:03,879 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:54:03,880 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-21 01:54:03,880 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:54:03,880 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:03,880 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:54:03,880 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:03,880 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:54:03,880 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:03,895 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=1218 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/bbe4322d6edc4c439e9cebc89b0604d2 2023-04-21 01:54:03,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:54:03,940 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:54:03,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1682042043947, max timestamp=1682042043965Num uniques 859; 2023-04-21 01:54:03,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-21 01:54:03,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1682042043947, max timestamp=1682042043965Num uniques 859; 2023-04-21 01:54:03,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-21 01:54:03,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1682042043947, max timestamp=1682042043965Num uniques 859; 2023-04-21 01:54:03,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-21 01:54:03,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:03,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042103984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:03,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:03,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 567 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042103990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:04,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042104091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:04,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 569 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042104093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,241 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/e7acf223542949499f77a66933777318 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/e7acf223542949499f77a66933777318 2023-04-21 01:54:04,243 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/8b250717ed7749d9a8063810584fa997 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/8b250717ed7749d9a8063810584fa997 2023-04-21 01:54:04,248 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/B of 16b5c71a3c47da32fe9f26ec9e41dfb8 into e7acf223542949499f77a66933777318(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:04,248 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:54:04,248 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/B, priority=13, startTime=1682042043792; duration=0sec 2023-04-21 01:54:04,248 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:04,248 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:04,253 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:04,253 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/C is initiating minor compaction (all files) 2023-04-21 01:54:04,253 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/C in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:04,253 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/8c5b991cb3bb4a9da9f4d2f1bbc7785d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/85f45fcf88af492981c753c8cde8ec08, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/32a9121b838f4df19dcfc4406961a7eb] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=36.9 K 2023-04-21 01:54:04,254 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 8c5b991cb3bb4a9da9f4d2f1bbc7785d, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1682042039474 2023-04-21 01:54:04,254 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 85f45fcf88af492981c753c8cde8ec08, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=793, earliestPutTs=1682042041297 2023-04-21 01:54:04,255 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 32a9121b838f4df19dcfc4406961a7eb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1209, earliestPutTs=1682042043301 2023-04-21 01:54:04,265 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/A of 16b5c71a3c47da32fe9f26ec9e41dfb8 into 8b250717ed7749d9a8063810584fa997(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:04,265 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:54:04,265 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/A, priority=13, startTime=1682042043792; duration=0sec 2023-04-21 01:54:04,265 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:04,276 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#C#compaction#125 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:04,293 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/ceb6890031944e09979996c987aa3241 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/ceb6890031944e09979996c987aa3241 2023-04-21 01:54:04,298 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/C of 16b5c71a3c47da32fe9f26ec9e41dfb8 into ceb6890031944e09979996c987aa3241(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:04,298 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:54:04,298 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/C, priority=13, startTime=1682042043792; duration=0sec 2023-04-21 01:54:04,299 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:04,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:04,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042104296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:04,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 571 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042104296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,314 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=1218 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/dc9dfed0d33c43b2846989d170e16d37 2023-04-21 01:54:04,326 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=1218 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/cd139cdca885482887e94e2121de40ce 2023-04-21 01:54:04,334 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/bbe4322d6edc4c439e9cebc89b0604d2 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/bbe4322d6edc4c439e9cebc89b0604d2 2023-04-21 01:54:04,340 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/bbe4322d6edc4c439e9cebc89b0604d2, entries=150, sequenceid=1218, filesize=11.9 K 2023-04-21 01:54:04,342 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/dc9dfed0d33c43b2846989d170e16d37 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/dc9dfed0d33c43b2846989d170e16d37 2023-04-21 01:54:04,346 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/dc9dfed0d33c43b2846989d170e16d37, entries=150, sequenceid=1218, filesize=11.9 K 2023-04-21 01:54:04,347 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/cd139cdca885482887e94e2121de40ce as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/cd139cdca885482887e94e2121de40ce 2023-04-21 01:54:04,352 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/cd139cdca885482887e94e2121de40ce, entries=150, sequenceid=1218, filesize=11.9 K 2023-04-21 01:54:04,353 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 474ms, sequenceid=1218, compaction requested=false 2023-04-21 01:54:04,353 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:54:04,353 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:04,353 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:04,354 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:04,354 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,355 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:04,356 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:04,357 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,357 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:04,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:04,358 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,358 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:04,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:04,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:04,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:04,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:04,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:04,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37c7f38d[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:04,359 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:04,359 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,361 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,361 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:04,361 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,361 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:04,361 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:04,361 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:04,364 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,364 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:04,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,364 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:04,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:04,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:04,364 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:04,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:04,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:04,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:04,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:04,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:04,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:04,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:04,366 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:04,366 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:04,366 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:04,366 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:04,366 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:04,368 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,368 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:04,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:04,368 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:04,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:04,368 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:04,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:04,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:04,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:04,369 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:04,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:04,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:04,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,372 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,372 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:04,372 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:04,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:04,372 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:04,372 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:04,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:04,372 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:04,372 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:04,372 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:04,372 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,372 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,372 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,372 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:04,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:04,372 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:04,372 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:04,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:04,472 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:04,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:04,475 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:04,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:04,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:04,484 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:04,484 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:04,484 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:04,484 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:04,485 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,485 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:04,486 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:04,486 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:04,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:04,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:04,487 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,487 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:04,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:04,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:04,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:04,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:04,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:04,494 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:04,494 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:04,495 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:04,496 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:04,496 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:04,496 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:04,496 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:54:04,496 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=147.60 KB heapSize=396.70 KB 2023-04-21 01:54:04,496 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:54:04,496 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:04,496 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:54:04,496 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:04,496 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:54:04,496 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:04,510 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=1254 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/3560f52604484437a12c9fc30cca1213 2023-04-21 01:54:04,533 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=1254 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/adb0f45b10cd4787a649d4bd4f7fc882 2023-04-21 01:54:04,555 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=1254 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/f5108ea736d74868a4d60aa6e9cba1c6 2023-04-21 01:54:04,560 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/3560f52604484437a12c9fc30cca1213 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/3560f52604484437a12c9fc30cca1213 2023-04-21 01:54:04,564 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/3560f52604484437a12c9fc30cca1213, entries=150, sequenceid=1254, filesize=11.9 K 2023-04-21 01:54:04,565 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/adb0f45b10cd4787a649d4bd4f7fc882 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/adb0f45b10cd4787a649d4bd4f7fc882 2023-04-21 01:54:04,569 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/adb0f45b10cd4787a649d4bd4f7fc882, entries=150, sequenceid=1254, filesize=11.9 K 2023-04-21 01:54:04,570 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/f5108ea736d74868a4d60aa6e9cba1c6 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/f5108ea736d74868a4d60aa6e9cba1c6 2023-04-21 01:54:04,574 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/f5108ea736d74868a4d60aa6e9cba1c6, entries=150, sequenceid=1254, filesize=11.9 K 2023-04-21 01:54:04,575 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~396.66 KB/406176, currentSize=0 B/0 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 79ms, sequenceid=1254, compaction requested=true 2023-04-21 01:54:04,575 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:54:04,575 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:04,575 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:04,576 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:04,576 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,576 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:04,576 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:04,580 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,580 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:04,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:04,580 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:04,580 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:04,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:04,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:04,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:04,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:04,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f14ae87[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:04,582 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:04,582 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,583 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,583 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:04,583 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:04,583 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:04,584 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,584 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:04,585 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:04,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:04,585 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:04,585 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:04,585 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:04,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:04,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:04,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:04,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:04,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:04,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:04,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:04,588 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:04,588 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:04,588 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:04,588 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:04,589 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:04,590 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,590 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,590 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:04,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:04,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:04,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:04,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:04,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:04,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,591 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:04,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:04,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:04,592 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:04,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:04,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,596 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,596 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:04,596 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:04,596 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:04,596 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,596 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:04,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:04,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:04,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:04,596 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:04,596 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:04,596 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:04,596 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,597 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,597 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:04,597 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:04,597 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:04,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:04,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:54:04,608 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-21 01:54:04,609 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:54:04,609 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:04,609 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:54:04,609 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:04,609 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:54:04,609 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:04,616 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1265 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/067127f3aa3147228715180364004923 2023-04-21 01:54:04,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682042044628, max timestamp=1682042044644Num uniques 859; 2023-04-21 01:54:04,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-21 01:54:04,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682042044628, max timestamp=1682042044644Num uniques 859; 2023-04-21 01:54:04,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-21 01:54:04,648 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682042044628, max timestamp=1682042044644Num uniques 859; 2023-04-21 01:54:04,648 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-21 01:54:04,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:04,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 427 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042104669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:04,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 593 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042104670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,697 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:04,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:04,699 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:04,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:04,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:04,701 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:04,701 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:04,701 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:04,701 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:04,701 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,701 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:04,703 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:04,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:04,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:04,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:04,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:04,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:04,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:04,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:04,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:04,706 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:04,706 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:04,706 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:04,706 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:04,706 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:04,706 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:04,706 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. started... 2023-04-21 01:54:04,707 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. as already flushing 2023-04-21 01:54:04,707 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:04,707 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:04,707 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:04,707 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,707 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:04,707 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:04,708 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,709 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,709 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:04,709 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cf512c9[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:04,709 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:04,709 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:04,709 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,709 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,709 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,709 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:04,714 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,714 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:04,714 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:04,714 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:04,715 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,715 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:04,716 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,716 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:04,716 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:04,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,716 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:04,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:04,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:04,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:04,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:04,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:04,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:04,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:04,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:04,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:04,719 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:04,719 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:04,719 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:04,719 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:04,719 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:04,721 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,721 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:04,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:04,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:04,722 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:04,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:04,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:04,723 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:04,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:04,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:04,725 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:04,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:04,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,728 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,728 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:04,728 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:04,728 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:04,728 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:04,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:04,728 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:04,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:04,728 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:04,728 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:04,729 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,729 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,729 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:04,729 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:04,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:04,729 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:04,729 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:04,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:04,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:04,771 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6aad2e18 to 127.0.0.1:58383 2023-04-21 01:54:04,771 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:54:04,772 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2879e2e1 to 127.0.0.1:58383 2023-04-21 01:54:04,772 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:54:04,772 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6fef7a21 to 127.0.0.1:58383 2023-04-21 01:54:04,773 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:54:04,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:04,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 429 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042104775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,777 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4bd6fc59 to 127.0.0.1:58383 2023-04-21 01:54:04,777 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:54:04,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:04,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 595 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042104778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,829 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:04,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:04,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:04,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 431 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042104977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:04,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:04,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 597 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042104979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:05,028 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1265 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/1430c0b4d53143539096d2608f2c2b5f 2023-04-21 01:54:05,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:05,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:05,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 599 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59312 deadline: 1682042105281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:05,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 433 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59292 deadline: 1682042105281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:05,439 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1265 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/b2336527f54743779a3110d5fb70057f 2023-04-21 01:54:05,444 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/067127f3aa3147228715180364004923 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/067127f3aa3147228715180364004923 2023-04-21 01:54:05,448 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/067127f3aa3147228715180364004923, entries=250, sequenceid=1265, filesize=16.7 K 2023-04-21 01:54:05,449 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/1430c0b4d53143539096d2608f2c2b5f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/1430c0b4d53143539096d2608f2c2b5f 2023-04-21 01:54:05,454 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/1430c0b4d53143539096d2608f2c2b5f, entries=150, sequenceid=1265, filesize=11.9 K 2023-04-21 01:54:05,454 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/b2336527f54743779a3110d5fb70057f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/b2336527f54743779a3110d5fb70057f 2023-04-21 01:54:05,459 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/b2336527f54743779a3110d5fb70057f, entries=150, sequenceid=1265, filesize=11.9 K 2023-04-21 01:54:05,459 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 851ms, sequenceid=1265, compaction requested=true 2023-04-21 01:54:05,460 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:54:05,460 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:05,460 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:05,460 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:54:05,460 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:54:05,460 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:05,461 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 55020 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:54:05,461 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50140 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:54:05,461 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/A is initiating minor compaction (all files) 2023-04-21 01:54:05,461 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/B is initiating minor compaction (all files) 2023-04-21 01:54:05,461 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/A in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:05,461 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/B in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:05,461 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/8b250717ed7749d9a8063810584fa997, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/bbe4322d6edc4c439e9cebc89b0604d2, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/3560f52604484437a12c9fc30cca1213, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/067127f3aa3147228715180364004923] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=53.7 K 2023-04-21 01:54:05,461 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/e7acf223542949499f77a66933777318, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/dc9dfed0d33c43b2846989d170e16d37, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/adb0f45b10cd4787a649d4bd4f7fc882, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/1430c0b4d53143539096d2608f2c2b5f] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=49.0 K 2023-04-21 01:54:05,462 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 8b250717ed7749d9a8063810584fa997, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=1209, earliestPutTs=1682042043301 2023-04-21 01:54:05,462 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting e7acf223542949499f77a66933777318, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=1209, earliestPutTs=1682042043301 2023-04-21 01:54:05,462 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting bbe4322d6edc4c439e9cebc89b0604d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1218, earliestPutTs=1682042043320 2023-04-21 01:54:05,462 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting dc9dfed0d33c43b2846989d170e16d37, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1218, earliestPutTs=1682042043320 2023-04-21 01:54:05,462 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 3560f52604484437a12c9fc30cca1213, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1254, earliestPutTs=1682042043961 2023-04-21 01:54:05,462 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting adb0f45b10cd4787a649d4bd4f7fc882, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1254, earliestPutTs=1682042043961 2023-04-21 01:54:05,463 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 067127f3aa3147228715180364004923, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=1265, earliestPutTs=1682042044602 2023-04-21 01:54:05,463 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 1430c0b4d53143539096d2608f2c2b5f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1265, earliestPutTs=1682042044605 2023-04-21 01:54:05,471 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#A#compaction#133 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:05,472 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#B#compaction#134 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:05,483 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/baa51b4015284798b58104cc14c280b0 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/baa51b4015284798b58104cc14c280b0 2023-04-21 01:54:05,488 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/B of 16b5c71a3c47da32fe9f26ec9e41dfb8 into baa51b4015284798b58104cc14c280b0(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:05,488 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:54:05,488 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/B, priority=12, startTime=1682042045460; duration=0sec 2023-04-21 01:54:05,488 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:05,488 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:54:05,489 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50072 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:54:05,490 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 16b5c71a3c47da32fe9f26ec9e41dfb8/C is initiating minor compaction (all files) 2023-04-21 01:54:05,490 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 16b5c71a3c47da32fe9f26ec9e41dfb8/C in TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:05,490 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/ceb6890031944e09979996c987aa3241, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/cd139cdca885482887e94e2121de40ce, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/f5108ea736d74868a4d60aa6e9cba1c6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/b2336527f54743779a3110d5fb70057f] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp, totalSize=48.9 K 2023-04-21 01:54:05,490 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting ceb6890031944e09979996c987aa3241, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=1209, earliestPutTs=1682042043301 2023-04-21 01:54:05,491 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting cd139cdca885482887e94e2121de40ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1218, earliestPutTs=1682042043320 2023-04-21 01:54:05,491 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting f5108ea736d74868a4d60aa6e9cba1c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1254, earliestPutTs=1682042043961 2023-04-21 01:54:05,491 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting b2336527f54743779a3110d5fb70057f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1265, earliestPutTs=1682042044605 2023-04-21 01:54:05,500 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 16b5c71a3c47da32fe9f26ec9e41dfb8#C#compaction#135 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:05,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:54:05,788 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-21 01:54:05,788 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x56c8e45e to 127.0.0.1:58383 2023-04-21 01:54:05,788 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:54:05,788 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0dbb9ac1 to 127.0.0.1:58383 2023-04-21 01:54:05,789 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:05,789 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:54:05,789 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:05,789 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:54:05,788 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:54:05,789 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:05,789 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:54:05,884 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/ce4e04ea43344ae49afcef0f34990e65 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/ce4e04ea43344ae49afcef0f34990e65 2023-04-21 01:54:05,889 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/A of 16b5c71a3c47da32fe9f26ec9e41dfb8 into ce4e04ea43344ae49afcef0f34990e65(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:05,889 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:54:05,889 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/A, priority=12, startTime=1682042045460; duration=0sec 2023-04-21 01:54:05,889 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:05,911 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/f2af9d6c9e994219862541c42980464e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/f2af9d6c9e994219862541c42980464e 2023-04-21 01:54:05,916 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 16b5c71a3c47da32fe9f26ec9e41dfb8/C of 16b5c71a3c47da32fe9f26ec9e41dfb8 into f2af9d6c9e994219862541c42980464e(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:05,916 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:54:05,916 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8., storeName=16b5c71a3c47da32fe9f26ec9e41dfb8/C, priority=12, startTime=1682042045460; duration=0sec 2023-04-21 01:54:05,916 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:06,205 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1302 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/90d65c60778f40769459eb3f62c30cd5 2023-04-21 01:54:06,630 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1302 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/4938149301af415c982ae166487dd06d 2023-04-21 01:54:07,044 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1302 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/9f8dbc6c5a7c4ffb98ecfc8f25a97fb2 2023-04-21 01:54:07,049 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/90d65c60778f40769459eb3f62c30cd5 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/90d65c60778f40769459eb3f62c30cd5 2023-04-21 01:54:07,052 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/90d65c60778f40769459eb3f62c30cd5, entries=150, sequenceid=1302, filesize=11.9 K 2023-04-21 01:54:07,053 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/4938149301af415c982ae166487dd06d as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/4938149301af415c982ae166487dd06d 2023-04-21 01:54:07,057 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/4938149301af415c982ae166487dd06d, entries=150, sequenceid=1302, filesize=11.9 K 2023-04-21 01:54:07,058 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/9f8dbc6c5a7c4ffb98ecfc8f25a97fb2 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/9f8dbc6c5a7c4ffb98ecfc8f25a97fb2 2023-04-21 01:54:07,061 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/9f8dbc6c5a7c4ffb98ecfc8f25a97fb2, entries=150, sequenceid=1302, filesize=11.9 K 2023-04-21 01:54:07,062 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=0 B/0 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 1274ms, sequenceid=1302, compaction requested=false 2023-04-21 01:54:07,062 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:54:13,686 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6038a026 to 127.0.0.1:58383 2023-04-21 01:54:13,686 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:54:13,704 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e91d243 to 127.0.0.1:58383 2023-04-21 01:54:13,704 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:54:13,713 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x46192d17 to 127.0.0.1:58383 2023-04-21 01:54:13,713 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:54:13,713 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-21 01:54:13,714 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 335 2023-04-21 01:54:13,714 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 206 2023-04-21 01:54:13,714 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 72 2023-04-21 01:54:13,714 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 85 2023-04-21 01:54:13,714 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 469 2023-04-21 01:54:13,714 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-21 01:54:13,714 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(398): read 3545 2023-04-21 01:54:13,714 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(398): read 3714 2023-04-21 01:54:13,714 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-21 01:54:13,714 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(402): scanned 1636 2023-04-21 01:54:13,714 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(403): verified 4893 rows 2023-04-21 01:54:13,714 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(402): scanned 1624 2023-04-21 01:54:13,715 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(403): verified 4863 rows 2023-04-21 01:54:13,715 INFO [Listener at localhost/39939] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-21 01:54:13,715 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b17e1c1 to 127.0.0.1:58383 2023-04-21 01:54:13,715 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:54:13,718 INFO [Listener at localhost/39939] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-21 01:54:13,725 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-21 01:54:13,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-21 01:54:13,735 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042053735"}]},"ts":"1682042053735"} 2023-04-21 01:54:13,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-21 01:54:13,736 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-21 01:54:13,818 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-21 01:54:13,821 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=16b5c71a3c47da32fe9f26ec9e41dfb8, UNASSIGN}] 2023-04-21 01:54:13,822 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=16b5c71a3c47da32fe9f26ec9e41dfb8, UNASSIGN 2023-04-21 01:54:13,822 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=16b5c71a3c47da32fe9f26ec9e41dfb8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:13,823 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042053822"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682042053822"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682042053822"}]},"ts":"1682042053822"} 2023-04-21 01:54:13,824 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399}] 2023-04-21 01:54:13,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-21 01:54:13,986 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:54:13,986 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 16b5c71a3c47da32fe9f26ec9e41dfb8, disabling compactions & flushes 2023-04-21 01:54:13,986 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:13,986 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:13,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. after waiting 0 ms 2023-04-21 01:54:13,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:13,987 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 16b5c71a3c47da32fe9f26ec9e41dfb8 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-21 01:54:13,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=A 2023-04-21 01:54:13,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:13,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=B 2023-04-21 01:54:13,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:13,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 16b5c71a3c47da32fe9f26ec9e41dfb8, store=C 2023-04-21 01:54:13,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:14,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-21 01:54:14,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-21 01:54:14,397 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1310 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/326de32aa9e1459ca3a57aa535129a0e 2023-04-21 01:54:14,816 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1310 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/51be1ccc33d54fa6825356b00301d0b8 2023-04-21 01:54:14,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-21 01:54:15,227 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1310 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/52c8782c074a4cc6ac4d15b1429c9d54 2023-04-21 01:54:15,233 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/A/326de32aa9e1459ca3a57aa535129a0e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/326de32aa9e1459ca3a57aa535129a0e 2023-04-21 01:54:15,238 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/326de32aa9e1459ca3a57aa535129a0e, entries=50, sequenceid=1310, filesize=7.1 K 2023-04-21 01:54:15,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/B/51be1ccc33d54fa6825356b00301d0b8 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/51be1ccc33d54fa6825356b00301d0b8 2023-04-21 01:54:15,244 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/51be1ccc33d54fa6825356b00301d0b8, entries=50, sequenceid=1310, filesize=7.1 K 2023-04-21 01:54:15,245 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/.tmp/C/52c8782c074a4cc6ac4d15b1429c9d54 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/52c8782c074a4cc6ac4d15b1429c9d54 2023-04-21 01:54:15,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/52c8782c074a4cc6ac4d15b1429c9d54, entries=50, sequenceid=1310, filesize=7.1 K 2023-04-21 01:54:15,251 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 16b5c71a3c47da32fe9f26ec9e41dfb8 in 1264ms, sequenceid=1310, compaction requested=true 2023-04-21 01:54:15,277 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/08ce249d075d43269941240c02e68dfc, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/9e710ac33917431fbd44470724b9dd43, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b5cd117cbbd84aa6a0ba94ab0c128ba8, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/dfc9ddd793b641ffb0a6bf196b91d94e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/0807cca5c9134179a1a71f07f50da02e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/44441447fd5146dcbbcb9448d88f959e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b905972c1fbd49a58a426eb0f7c29905, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/4a48a64bba844b598c56189250f3569f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/adb5a5aa3a2f4e89b034fd704b422996, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b801f927d9034a36905920a2d9ad0ab0, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/39d7407cc7c44e7fb1f5ad3907f7bfaa, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/f4dc85d78cf34cb881f1dc540f1b0d33, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/c6970e5e10bb43ab9e249f51377cabef, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/94b78c7f9692459d97a3badd102551db, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/ac38b0648e1c4dd29bd7e4594c2f5f35, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/cc504ad8b2df41a9bfee6a0c011c8b10, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/7536aa07d5ed4c5c8c9f666b72289e66, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/9f3dcd3239cd44e9bcc883abddbd8a0d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/cc246645a35449a7a9e27cf9e0e56fec, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/320f4a48849d475d99606bc7541b6ce6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/785fbb29ac774d1582d0ba5fcf0f0b9c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/04eb6ec9a26e4dcc8db5b734c493a2f7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b6c1eef51bf548bb8db8baf2edd23e88, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/f356b223339d44bb88c85ea96a6d5e9c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/77557505b95144e68b918440756fead2, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/ce471700f59848e6bfc34596cec60432, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/3b7ebdf308fa48a2acdd6bab98605adf, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/576f481dfad24e06bdc41efedf5fc28f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/058465b805c84962aa48f55fc2045356, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/6cc0e4ec58724f5e8e1e81abbbbfd628, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/faa8d302554b496f9a64ca2dc90b05f9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/cc4d1b2c5c3348ed900fce7e7ed285ef, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/371dff0409574db2a0dd82d9ccb99c8e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/5fc1418c43774fc2bb9f494301d8430c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b6ef1e9c01e04cc688259ed3db337e29, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/e77f1ea4d6d149639016b19052450f93, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/2958825b35704d6e86b71f55daf652d1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/2f62574fceaa4234bec2dc8c2a830b90, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/6bca07c3085b4be48b0c5bf2bed2a858, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/5022dc45ced2448a9d8dbd791bcfc4d5, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/96e871f9f9204bbe89a3cd8eaf4c3a16, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/8b250717ed7749d9a8063810584fa997, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/bbe4322d6edc4c439e9cebc89b0604d2, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/3560f52604484437a12c9fc30cca1213, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/067127f3aa3147228715180364004923] to archive 2023-04-21 01:54:15,279 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-21 01:54:15,287 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/08ce249d075d43269941240c02e68dfc to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/08ce249d075d43269941240c02e68dfc 2023-04-21 01:54:15,288 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/9e710ac33917431fbd44470724b9dd43 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/9e710ac33917431fbd44470724b9dd43 2023-04-21 01:54:15,290 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b5cd117cbbd84aa6a0ba94ab0c128ba8 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b5cd117cbbd84aa6a0ba94ab0c128ba8 2023-04-21 01:54:15,291 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/dfc9ddd793b641ffb0a6bf196b91d94e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/dfc9ddd793b641ffb0a6bf196b91d94e 2023-04-21 01:54:15,292 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/0807cca5c9134179a1a71f07f50da02e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/0807cca5c9134179a1a71f07f50da02e 2023-04-21 01:54:15,294 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/44441447fd5146dcbbcb9448d88f959e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/44441447fd5146dcbbcb9448d88f959e 2023-04-21 01:54:15,295 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b905972c1fbd49a58a426eb0f7c29905 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b905972c1fbd49a58a426eb0f7c29905 2023-04-21 01:54:15,297 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/4a48a64bba844b598c56189250f3569f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/4a48a64bba844b598c56189250f3569f 2023-04-21 01:54:15,298 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/adb5a5aa3a2f4e89b034fd704b422996 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/adb5a5aa3a2f4e89b034fd704b422996 2023-04-21 01:54:15,299 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b801f927d9034a36905920a2d9ad0ab0 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b801f927d9034a36905920a2d9ad0ab0 2023-04-21 01:54:15,300 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/39d7407cc7c44e7fb1f5ad3907f7bfaa to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/39d7407cc7c44e7fb1f5ad3907f7bfaa 2023-04-21 01:54:15,301 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/f4dc85d78cf34cb881f1dc540f1b0d33 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/f4dc85d78cf34cb881f1dc540f1b0d33 2023-04-21 01:54:15,302 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/c6970e5e10bb43ab9e249f51377cabef to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/c6970e5e10bb43ab9e249f51377cabef 2023-04-21 01:54:15,303 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/94b78c7f9692459d97a3badd102551db to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/94b78c7f9692459d97a3badd102551db 2023-04-21 01:54:15,305 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/ac38b0648e1c4dd29bd7e4594c2f5f35 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/ac38b0648e1c4dd29bd7e4594c2f5f35 2023-04-21 01:54:15,307 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/cc504ad8b2df41a9bfee6a0c011c8b10 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/cc504ad8b2df41a9bfee6a0c011c8b10 2023-04-21 01:54:15,308 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/7536aa07d5ed4c5c8c9f666b72289e66 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/7536aa07d5ed4c5c8c9f666b72289e66 2023-04-21 01:54:15,309 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/9f3dcd3239cd44e9bcc883abddbd8a0d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/9f3dcd3239cd44e9bcc883abddbd8a0d 2023-04-21 01:54:15,310 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/cc246645a35449a7a9e27cf9e0e56fec to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/cc246645a35449a7a9e27cf9e0e56fec 2023-04-21 01:54:15,311 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/320f4a48849d475d99606bc7541b6ce6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/320f4a48849d475d99606bc7541b6ce6 2023-04-21 01:54:15,312 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/785fbb29ac774d1582d0ba5fcf0f0b9c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/785fbb29ac774d1582d0ba5fcf0f0b9c 2023-04-21 01:54:15,314 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/04eb6ec9a26e4dcc8db5b734c493a2f7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/04eb6ec9a26e4dcc8db5b734c493a2f7 2023-04-21 01:54:15,315 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b6c1eef51bf548bb8db8baf2edd23e88 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b6c1eef51bf548bb8db8baf2edd23e88 2023-04-21 01:54:15,316 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/f356b223339d44bb88c85ea96a6d5e9c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/f356b223339d44bb88c85ea96a6d5e9c 2023-04-21 01:54:15,317 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/77557505b95144e68b918440756fead2 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/77557505b95144e68b918440756fead2 2023-04-21 01:54:15,318 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/ce471700f59848e6bfc34596cec60432 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/ce471700f59848e6bfc34596cec60432 2023-04-21 01:54:15,319 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/3b7ebdf308fa48a2acdd6bab98605adf to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/3b7ebdf308fa48a2acdd6bab98605adf 2023-04-21 01:54:15,320 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/576f481dfad24e06bdc41efedf5fc28f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/576f481dfad24e06bdc41efedf5fc28f 2023-04-21 01:54:15,321 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/058465b805c84962aa48f55fc2045356 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/058465b805c84962aa48f55fc2045356 2023-04-21 01:54:15,322 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/6cc0e4ec58724f5e8e1e81abbbbfd628 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/6cc0e4ec58724f5e8e1e81abbbbfd628 2023-04-21 01:54:15,323 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/faa8d302554b496f9a64ca2dc90b05f9 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/faa8d302554b496f9a64ca2dc90b05f9 2023-04-21 01:54:15,325 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/cc4d1b2c5c3348ed900fce7e7ed285ef to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/cc4d1b2c5c3348ed900fce7e7ed285ef 2023-04-21 01:54:15,326 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/371dff0409574db2a0dd82d9ccb99c8e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/371dff0409574db2a0dd82d9ccb99c8e 2023-04-21 01:54:15,327 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/5fc1418c43774fc2bb9f494301d8430c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/5fc1418c43774fc2bb9f494301d8430c 2023-04-21 01:54:15,328 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b6ef1e9c01e04cc688259ed3db337e29 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/b6ef1e9c01e04cc688259ed3db337e29 2023-04-21 01:54:15,329 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/e77f1ea4d6d149639016b19052450f93 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/e77f1ea4d6d149639016b19052450f93 2023-04-21 01:54:15,330 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/2958825b35704d6e86b71f55daf652d1 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/2958825b35704d6e86b71f55daf652d1 2023-04-21 01:54:15,331 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/2f62574fceaa4234bec2dc8c2a830b90 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/2f62574fceaa4234bec2dc8c2a830b90 2023-04-21 01:54:15,332 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/6bca07c3085b4be48b0c5bf2bed2a858 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/6bca07c3085b4be48b0c5bf2bed2a858 2023-04-21 01:54:15,333 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/5022dc45ced2448a9d8dbd791bcfc4d5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/5022dc45ced2448a9d8dbd791bcfc4d5 2023-04-21 01:54:15,334 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/96e871f9f9204bbe89a3cd8eaf4c3a16 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/96e871f9f9204bbe89a3cd8eaf4c3a16 2023-04-21 01:54:15,335 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/8b250717ed7749d9a8063810584fa997 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/8b250717ed7749d9a8063810584fa997 2023-04-21 01:54:15,336 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/bbe4322d6edc4c439e9cebc89b0604d2 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/bbe4322d6edc4c439e9cebc89b0604d2 2023-04-21 01:54:15,337 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/3560f52604484437a12c9fc30cca1213 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/3560f52604484437a12c9fc30cca1213 2023-04-21 01:54:15,338 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/067127f3aa3147228715180364004923 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/067127f3aa3147228715180364004923 2023-04-21 01:54:15,364 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/3f82bce1e29c4cd98b5c58d26860d931, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/17f1d4a52e2047e3a7191a0ca78d4b70, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/cb2aa42de24747de9fd3cc3473ec63ee, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/8cbcaf0f89b74b91a066d395b5c41bb2, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/30e2da97fac6451a9247350c25b12b6a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/de11981e03ea4ebc9d406480c7578199, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/a59cca98e04641ad955f732fbe59657c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/7a749b8b1c274a979edfab794e5df3dd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/126287b21da74ddd8d9f1e7bc10f438e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/b7c969d50f884cf4bf34a136a6d8eb73, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/dad73e23304c4dce81ba08091d21fdde, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/2fd310eb7c6b43a2994cf0c995750338, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/b3acadcd4aa34e4f9e821137527a401f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/cb8e7b80d26849618cdac024eb31921b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/51f172d9138544a3a80f2f173a464790, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/3de31e6e4a464b0aa053849e29f03206, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/244c34418771484b926c51928969a40a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/ae8007396436475eab082aef1d03a7ea, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/06731cb476b646879256f4cd176ca112, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/834793b7ccd4430596eea3cbf5ccfc57, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/903002e959ab458a837bcdfc33029b95, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/a043d0e57a964a42b4cbefa8454c5bb1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/321d5f27a0664aceb223a84f43cf6444, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/0b3d26375ce14c4b9f4257c270e08119, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/d0fe3d88c2df4f9e9d9af3b41e811848, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/01ecaebb7f104f079d1e6b021f25fa36, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/52cf492a4a6048ddb59c7c3425db752e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/8f153e92b56d458a8fe901886fa79703, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/e714ca69d7e14c0a899cf95a4029cfd7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/22591c4b86844ea6916884d5334b9e1e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/0217985618e346fca914f671a8e52296, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/c4fda86e7b5e4572b63d4aab8c27d0e0, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/4a98698cedd84d9b9d89bb184450bd3f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/cd2be2802e0848f4b0a80ee17485e5d1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/833099ab51cb45f084359b5cf9e69a99, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/f4884a0307c74f38821cd457d18eed6c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/d952a1a00b60435f88b533935df1abdb, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/9410f60fc7244e689a4f385bf03dc7e7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/f62de8a1a68d43038645a8aa16e1b039, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/a96e214169a24170b436eb41eddb503e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/e7acf223542949499f77a66933777318, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/b2f082f8c24441daa681eb91eb168b35, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/dc9dfed0d33c43b2846989d170e16d37, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/adb0f45b10cd4787a649d4bd4f7fc882, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/1430c0b4d53143539096d2608f2c2b5f] to archive 2023-04-21 01:54:15,365 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-21 01:54:15,367 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/3f82bce1e29c4cd98b5c58d26860d931 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/3f82bce1e29c4cd98b5c58d26860d931 2023-04-21 01:54:15,368 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/17f1d4a52e2047e3a7191a0ca78d4b70 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/17f1d4a52e2047e3a7191a0ca78d4b70 2023-04-21 01:54:15,369 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/cb2aa42de24747de9fd3cc3473ec63ee to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/cb2aa42de24747de9fd3cc3473ec63ee 2023-04-21 01:54:15,370 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/8cbcaf0f89b74b91a066d395b5c41bb2 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/8cbcaf0f89b74b91a066d395b5c41bb2 2023-04-21 01:54:15,371 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/30e2da97fac6451a9247350c25b12b6a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/30e2da97fac6451a9247350c25b12b6a 2023-04-21 01:54:15,372 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/de11981e03ea4ebc9d406480c7578199 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/de11981e03ea4ebc9d406480c7578199 2023-04-21 01:54:15,373 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/a59cca98e04641ad955f732fbe59657c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/a59cca98e04641ad955f732fbe59657c 2023-04-21 01:54:15,374 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/7a749b8b1c274a979edfab794e5df3dd to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/7a749b8b1c274a979edfab794e5df3dd 2023-04-21 01:54:15,375 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/126287b21da74ddd8d9f1e7bc10f438e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/126287b21da74ddd8d9f1e7bc10f438e 2023-04-21 01:54:15,376 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/b7c969d50f884cf4bf34a136a6d8eb73 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/b7c969d50f884cf4bf34a136a6d8eb73 2023-04-21 01:54:15,377 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/dad73e23304c4dce81ba08091d21fdde to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/dad73e23304c4dce81ba08091d21fdde 2023-04-21 01:54:15,378 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/2fd310eb7c6b43a2994cf0c995750338 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/2fd310eb7c6b43a2994cf0c995750338 2023-04-21 01:54:15,379 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/b3acadcd4aa34e4f9e821137527a401f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/b3acadcd4aa34e4f9e821137527a401f 2023-04-21 01:54:15,380 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/cb8e7b80d26849618cdac024eb31921b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/cb8e7b80d26849618cdac024eb31921b 2023-04-21 01:54:15,382 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/51f172d9138544a3a80f2f173a464790 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/51f172d9138544a3a80f2f173a464790 2023-04-21 01:54:15,383 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/3de31e6e4a464b0aa053849e29f03206 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/3de31e6e4a464b0aa053849e29f03206 2023-04-21 01:54:15,384 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/244c34418771484b926c51928969a40a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/244c34418771484b926c51928969a40a 2023-04-21 01:54:15,385 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/ae8007396436475eab082aef1d03a7ea to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/ae8007396436475eab082aef1d03a7ea 2023-04-21 01:54:15,386 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/06731cb476b646879256f4cd176ca112 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/06731cb476b646879256f4cd176ca112 2023-04-21 01:54:15,387 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/834793b7ccd4430596eea3cbf5ccfc57 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/834793b7ccd4430596eea3cbf5ccfc57 2023-04-21 01:54:15,388 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/903002e959ab458a837bcdfc33029b95 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/903002e959ab458a837bcdfc33029b95 2023-04-21 01:54:15,390 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/a043d0e57a964a42b4cbefa8454c5bb1 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/a043d0e57a964a42b4cbefa8454c5bb1 2023-04-21 01:54:15,391 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/321d5f27a0664aceb223a84f43cf6444 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/321d5f27a0664aceb223a84f43cf6444 2023-04-21 01:54:15,392 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/0b3d26375ce14c4b9f4257c270e08119 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/0b3d26375ce14c4b9f4257c270e08119 2023-04-21 01:54:15,393 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/d0fe3d88c2df4f9e9d9af3b41e811848 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/d0fe3d88c2df4f9e9d9af3b41e811848 2023-04-21 01:54:15,394 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/01ecaebb7f104f079d1e6b021f25fa36 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/01ecaebb7f104f079d1e6b021f25fa36 2023-04-21 01:54:15,395 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/52cf492a4a6048ddb59c7c3425db752e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/52cf492a4a6048ddb59c7c3425db752e 2023-04-21 01:54:15,396 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/8f153e92b56d458a8fe901886fa79703 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/8f153e92b56d458a8fe901886fa79703 2023-04-21 01:54:15,397 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/e714ca69d7e14c0a899cf95a4029cfd7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/e714ca69d7e14c0a899cf95a4029cfd7 2023-04-21 01:54:15,398 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/22591c4b86844ea6916884d5334b9e1e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/22591c4b86844ea6916884d5334b9e1e 2023-04-21 01:54:15,399 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/0217985618e346fca914f671a8e52296 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/0217985618e346fca914f671a8e52296 2023-04-21 01:54:15,400 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/c4fda86e7b5e4572b63d4aab8c27d0e0 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/c4fda86e7b5e4572b63d4aab8c27d0e0 2023-04-21 01:54:15,402 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/4a98698cedd84d9b9d89bb184450bd3f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/4a98698cedd84d9b9d89bb184450bd3f 2023-04-21 01:54:15,403 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/cd2be2802e0848f4b0a80ee17485e5d1 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/cd2be2802e0848f4b0a80ee17485e5d1 2023-04-21 01:54:15,404 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/833099ab51cb45f084359b5cf9e69a99 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/833099ab51cb45f084359b5cf9e69a99 2023-04-21 01:54:15,405 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/f4884a0307c74f38821cd457d18eed6c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/f4884a0307c74f38821cd457d18eed6c 2023-04-21 01:54:15,406 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/d952a1a00b60435f88b533935df1abdb to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/d952a1a00b60435f88b533935df1abdb 2023-04-21 01:54:15,407 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/9410f60fc7244e689a4f385bf03dc7e7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/9410f60fc7244e689a4f385bf03dc7e7 2023-04-21 01:54:15,408 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/f62de8a1a68d43038645a8aa16e1b039 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/f62de8a1a68d43038645a8aa16e1b039 2023-04-21 01:54:15,410 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/a96e214169a24170b436eb41eddb503e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/a96e214169a24170b436eb41eddb503e 2023-04-21 01:54:15,411 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/e7acf223542949499f77a66933777318 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/e7acf223542949499f77a66933777318 2023-04-21 01:54:15,412 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/b2f082f8c24441daa681eb91eb168b35 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/b2f082f8c24441daa681eb91eb168b35 2023-04-21 01:54:15,413 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/dc9dfed0d33c43b2846989d170e16d37 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/dc9dfed0d33c43b2846989d170e16d37 2023-04-21 01:54:15,414 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/adb0f45b10cd4787a649d4bd4f7fc882 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/adb0f45b10cd4787a649d4bd4f7fc882 2023-04-21 01:54:15,416 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/1430c0b4d53143539096d2608f2c2b5f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/1430c0b4d53143539096d2608f2c2b5f 2023-04-21 01:54:15,420 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/d14d111773c94bfda63e4ae73c42b364, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/2d8420ee76ac438aa3b8a6fdffe5d901, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/c3fe54704263476cb315207d4e23c1c6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/8333f0b9c3c04d4ca22ee1d56f706828, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/c500a9a9d8fa40f089d191b93770dcc1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/408896e8c85a49a5a55e1d2fad7e42d9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/eecbfb279e6644898a87caa604a1c345, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/6d4e5f6783174912a7fe06b88da34d9b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/2a32e09968d84d61b2c597c3e65f7f03, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/c0d44205feb04bc58e7d3476dac1014c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/57be0fc4b4d44ec68ee2b11c44790d72, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/ff700b092a124cbfbbd6b55cd62960ec, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/a04c2bacacae48e5bf4a47ed17c88fea, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/6a243df1b5dd4e06b55f7ad64e7b2768, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/9404233302cd4638998c17b3a5260dad, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/fad0173f016544eca87ee7d593ddeb8f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/f26105d4af68439ba4bcebacb3a886ce, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/78542f67720249dcb31e3c798ded7510, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/8e387eee7fd8445cb664c04937f1d3a7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/2393e1ab44a54a09b45fc569a6bb0d38, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/5a31cf5e56a64e62a72dffb0745b6aa9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/67f97166b43b4d2088c17c29001996bb, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/1adef58ece594d42adde12f062976884, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/bae4038ed4bd4d99a16c3538ea3f526b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/a57ce6ca99c140c4a0a9fb064e9bc739, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/b5ba3d6b5f9b44e7ad5fc46527804c21, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/4fad11ba83554c0f908962810a493969, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/d9e11af48b9547d69afbaec8029b5244, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/dd7e53da8a05472884294e290ce259ac, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/9a865824f87b4133a05cee5cdfca8921, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/6aa9e03a9a5b4e5c894ef500de98e268, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/8e33c09a45b942fba011cf231def2fa8, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/760d3e13a6074c6abc6f8222d0382a6a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/c2923e7aca374fa18622bd11e0b55444, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/f09e3a15bed6458aa94c1532c511a18f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/8c5b991cb3bb4a9da9f4d2f1bbc7785d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/28e0e530c8624ac8b4e92d021226f2d7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/85f45fcf88af492981c753c8cde8ec08, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/ceb6890031944e09979996c987aa3241, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/32a9121b838f4df19dcfc4406961a7eb, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/cd139cdca885482887e94e2121de40ce, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/f5108ea736d74868a4d60aa6e9cba1c6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/b2336527f54743779a3110d5fb70057f] to archive 2023-04-21 01:54:15,421 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-21 01:54:15,423 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/d14d111773c94bfda63e4ae73c42b364 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/d14d111773c94bfda63e4ae73c42b364 2023-04-21 01:54:15,424 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/2d8420ee76ac438aa3b8a6fdffe5d901 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/2d8420ee76ac438aa3b8a6fdffe5d901 2023-04-21 01:54:15,425 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/c3fe54704263476cb315207d4e23c1c6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/c3fe54704263476cb315207d4e23c1c6 2023-04-21 01:54:15,426 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/8333f0b9c3c04d4ca22ee1d56f706828 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/8333f0b9c3c04d4ca22ee1d56f706828 2023-04-21 01:54:15,427 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/c500a9a9d8fa40f089d191b93770dcc1 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/c500a9a9d8fa40f089d191b93770dcc1 2023-04-21 01:54:15,428 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/408896e8c85a49a5a55e1d2fad7e42d9 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/408896e8c85a49a5a55e1d2fad7e42d9 2023-04-21 01:54:15,429 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/eecbfb279e6644898a87caa604a1c345 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/eecbfb279e6644898a87caa604a1c345 2023-04-21 01:54:15,430 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/6d4e5f6783174912a7fe06b88da34d9b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/6d4e5f6783174912a7fe06b88da34d9b 2023-04-21 01:54:15,431 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/2a32e09968d84d61b2c597c3e65f7f03 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/2a32e09968d84d61b2c597c3e65f7f03 2023-04-21 01:54:15,432 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/c0d44205feb04bc58e7d3476dac1014c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/c0d44205feb04bc58e7d3476dac1014c 2023-04-21 01:54:15,433 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/57be0fc4b4d44ec68ee2b11c44790d72 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/57be0fc4b4d44ec68ee2b11c44790d72 2023-04-21 01:54:15,435 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/ff700b092a124cbfbbd6b55cd62960ec to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/ff700b092a124cbfbbd6b55cd62960ec 2023-04-21 01:54:15,436 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/a04c2bacacae48e5bf4a47ed17c88fea to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/a04c2bacacae48e5bf4a47ed17c88fea 2023-04-21 01:54:15,437 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/6a243df1b5dd4e06b55f7ad64e7b2768 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/6a243df1b5dd4e06b55f7ad64e7b2768 2023-04-21 01:54:15,438 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/9404233302cd4638998c17b3a5260dad to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/9404233302cd4638998c17b3a5260dad 2023-04-21 01:54:15,440 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/fad0173f016544eca87ee7d593ddeb8f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/fad0173f016544eca87ee7d593ddeb8f 2023-04-21 01:54:15,441 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/f26105d4af68439ba4bcebacb3a886ce to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/f26105d4af68439ba4bcebacb3a886ce 2023-04-21 01:54:15,442 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/78542f67720249dcb31e3c798ded7510 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/78542f67720249dcb31e3c798ded7510 2023-04-21 01:54:15,444 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/8e387eee7fd8445cb664c04937f1d3a7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/8e387eee7fd8445cb664c04937f1d3a7 2023-04-21 01:54:15,446 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/2393e1ab44a54a09b45fc569a6bb0d38 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/2393e1ab44a54a09b45fc569a6bb0d38 2023-04-21 01:54:15,447 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/5a31cf5e56a64e62a72dffb0745b6aa9 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/5a31cf5e56a64e62a72dffb0745b6aa9 2023-04-21 01:54:15,448 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/67f97166b43b4d2088c17c29001996bb to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/67f97166b43b4d2088c17c29001996bb 2023-04-21 01:54:15,449 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/1adef58ece594d42adde12f062976884 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/1adef58ece594d42adde12f062976884 2023-04-21 01:54:15,450 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/bae4038ed4bd4d99a16c3538ea3f526b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/bae4038ed4bd4d99a16c3538ea3f526b 2023-04-21 01:54:15,451 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/a57ce6ca99c140c4a0a9fb064e9bc739 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/a57ce6ca99c140c4a0a9fb064e9bc739 2023-04-21 01:54:15,452 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/b5ba3d6b5f9b44e7ad5fc46527804c21 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/b5ba3d6b5f9b44e7ad5fc46527804c21 2023-04-21 01:54:15,453 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/4fad11ba83554c0f908962810a493969 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/4fad11ba83554c0f908962810a493969 2023-04-21 01:54:15,454 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/d9e11af48b9547d69afbaec8029b5244 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/d9e11af48b9547d69afbaec8029b5244 2023-04-21 01:54:15,456 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/dd7e53da8a05472884294e290ce259ac to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/dd7e53da8a05472884294e290ce259ac 2023-04-21 01:54:15,457 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/9a865824f87b4133a05cee5cdfca8921 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/9a865824f87b4133a05cee5cdfca8921 2023-04-21 01:54:15,458 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/6aa9e03a9a5b4e5c894ef500de98e268 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/6aa9e03a9a5b4e5c894ef500de98e268 2023-04-21 01:54:15,459 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/8e33c09a45b942fba011cf231def2fa8 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/8e33c09a45b942fba011cf231def2fa8 2023-04-21 01:54:15,460 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/760d3e13a6074c6abc6f8222d0382a6a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/760d3e13a6074c6abc6f8222d0382a6a 2023-04-21 01:54:15,461 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/c2923e7aca374fa18622bd11e0b55444 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/c2923e7aca374fa18622bd11e0b55444 2023-04-21 01:54:15,462 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/f09e3a15bed6458aa94c1532c511a18f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/f09e3a15bed6458aa94c1532c511a18f 2023-04-21 01:54:15,463 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/8c5b991cb3bb4a9da9f4d2f1bbc7785d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/8c5b991cb3bb4a9da9f4d2f1bbc7785d 2023-04-21 01:54:15,464 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/28e0e530c8624ac8b4e92d021226f2d7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/28e0e530c8624ac8b4e92d021226f2d7 2023-04-21 01:54:15,465 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/85f45fcf88af492981c753c8cde8ec08 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/85f45fcf88af492981c753c8cde8ec08 2023-04-21 01:54:15,466 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/ceb6890031944e09979996c987aa3241 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/ceb6890031944e09979996c987aa3241 2023-04-21 01:54:15,468 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/32a9121b838f4df19dcfc4406961a7eb to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/32a9121b838f4df19dcfc4406961a7eb 2023-04-21 01:54:15,469 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/cd139cdca885482887e94e2121de40ce to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/cd139cdca885482887e94e2121de40ce 2023-04-21 01:54:15,470 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/f5108ea736d74868a4d60aa6e9cba1c6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/f5108ea736d74868a4d60aa6e9cba1c6 2023-04-21 01:54:15,471 DEBUG [StoreCloser-TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/b2336527f54743779a3110d5fb70057f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/b2336527f54743779a3110d5fb70057f 2023-04-21 01:54:15,476 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/recovered.edits/1313.seqid, newMaxSeqId=1313, maxSeqId=1 2023-04-21 01:54:15,478 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8. 2023-04-21 01:54:15,478 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 16b5c71a3c47da32fe9f26ec9e41dfb8: 2023-04-21 01:54:15,479 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:54:15,480 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=16b5c71a3c47da32fe9f26ec9e41dfb8, regionState=CLOSED 2023-04-21 01:54:15,480 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042055480"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682042055480"}]},"ts":"1682042055480"} 2023-04-21 01:54:15,482 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-21 01:54:15,483 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 16b5c71a3c47da32fe9f26ec9e41dfb8, server=jenkins-hbase4.apache.org,40131,1682042019399 in 1.6570 sec 2023-04-21 01:54:15,484 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-21 01:54:15,484 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=16b5c71a3c47da32fe9f26ec9e41dfb8, UNASSIGN in 1.6630 sec 2023-04-21 01:54:15,485 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042055484"}]},"ts":"1682042055484"} 2023-04-21 01:54:15,486 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-21 01:54:15,487 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-21 01:54:15,489 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.7610 sec 2023-04-21 01:54:15,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-21 01:54:15,841 INFO [Listener at localhost/39939] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-21 01:54:15,845 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-21 01:54:15,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:54:15,853 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:54:15,855 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:54:15,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-21 01:54:15,859 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:54:15,862 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A, FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B, FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C, FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/recovered.edits] 2023-04-21 01:54:15,865 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/326de32aa9e1459ca3a57aa535129a0e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/326de32aa9e1459ca3a57aa535129a0e 2023-04-21 01:54:15,866 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/90d65c60778f40769459eb3f62c30cd5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/90d65c60778f40769459eb3f62c30cd5 2023-04-21 01:54:15,867 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/ce4e04ea43344ae49afcef0f34990e65 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/A/ce4e04ea43344ae49afcef0f34990e65 2023-04-21 01:54:15,870 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/4938149301af415c982ae166487dd06d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/4938149301af415c982ae166487dd06d 2023-04-21 01:54:15,871 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/51be1ccc33d54fa6825356b00301d0b8 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/51be1ccc33d54fa6825356b00301d0b8 2023-04-21 01:54:15,872 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/baa51b4015284798b58104cc14c280b0 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/B/baa51b4015284798b58104cc14c280b0 2023-04-21 01:54:15,874 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/52c8782c074a4cc6ac4d15b1429c9d54 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/52c8782c074a4cc6ac4d15b1429c9d54 2023-04-21 01:54:15,875 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/9f8dbc6c5a7c4ffb98ecfc8f25a97fb2 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/9f8dbc6c5a7c4ffb98ecfc8f25a97fb2 2023-04-21 01:54:15,876 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/f2af9d6c9e994219862541c42980464e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/C/f2af9d6c9e994219862541c42980464e 2023-04-21 01:54:15,879 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/recovered.edits/1313.seqid to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8/recovered.edits/1313.seqid 2023-04-21 01:54:15,880 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/16b5c71a3c47da32fe9f26ec9e41dfb8 2023-04-21 01:54:15,880 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-21 01:54:15,882 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:54:15,903 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-21 01:54:15,948 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-21 01:54:15,949 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:54:15,949 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-21 01:54:15,949 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682042055949"}]},"ts":"9223372036854775807"} 2023-04-21 01:54:15,952 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-21 01:54:15,952 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 16b5c71a3c47da32fe9f26ec9e41dfb8, NAME => 'TestAcidGuarantees,,1682042023463.16b5c71a3c47da32fe9f26ec9e41dfb8.', STARTKEY => '', ENDKEY => ''}] 2023-04-21 01:54:15,952 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-21 01:54:15,952 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682042055952"}]},"ts":"9223372036854775807"} 2023-04-21 01:54:15,954 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-21 01:54:15,956 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:54:15,957 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 110 msec 2023-04-21 01:54:15,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-21 01:54:15,957 INFO [Listener at localhost/39939] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-21 01:54:15,974 INFO [Listener at localhost/39939] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=229 (was 205) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4dbb6610-shared-pool-24 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1450051282_15 at /127.0.0.1:59846 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4dbb6610-shared-pool-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4dbb6610-shared-pool-26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:40131-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4dbb6610-shared-pool-23 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=560 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=509 (was 444) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=8865 (was 9540) 2023-04-21 01:54:15,985 INFO [Listener at localhost/39939] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=229, OpenFileDescriptor=560, MaxFileDescriptor=60000, SystemLoadAverage=509, ProcessCount=169, AvailableMemoryMB=8864 2023-04-21 01:54:15,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-21 01:54:15,987 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-21 01:54:15,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-21 01:54:15,990 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-21 01:54:15,990 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-21 01:54:15,990 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-21 01:54:15,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-21 01:54:15,992 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:15,993 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662 empty. 2023-04-21 01:54:15,993 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:15,993 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-21 01:54:16,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-21 01:54:16,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-21 01:54:16,402 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-21 01:54:16,403 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => e00828c94a25954f7bdc6de3ac3e2662, NAME => 'TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp 2023-04-21 01:54:16,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-21 01:54:16,810 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-21 01:54:16,810 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing e00828c94a25954f7bdc6de3ac3e2662, disabling compactions & flushes 2023-04-21 01:54:16,810 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:16,810 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:16,810 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. after waiting 0 ms 2023-04-21 01:54:16,810 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:16,810 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:16,810 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:16,812 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-21 01:54:16,813 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042056813"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682042056813"}]},"ts":"1682042056813"} 2023-04-21 01:54:16,815 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-21 01:54:16,815 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-21 01:54:16,816 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042056816"}]},"ts":"1682042056816"} 2023-04-21 01:54:16,817 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-21 01:54:16,819 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e00828c94a25954f7bdc6de3ac3e2662, ASSIGN}] 2023-04-21 01:54:16,821 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e00828c94a25954f7bdc6de3ac3e2662, ASSIGN 2023-04-21 01:54:16,821 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e00828c94a25954f7bdc6de3ac3e2662, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40131,1682042019399; forceNewPlan=false, retain=false 2023-04-21 01:54:16,973 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=e00828c94a25954f7bdc6de3ac3e2662, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:16,973 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042056972"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682042056972"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682042056972"}]},"ts":"1682042056972"} 2023-04-21 01:54:16,975 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399}] 2023-04-21 01:54:17,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-21 01:54:17,130 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:17,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e00828c94a25954f7bdc6de3ac3e2662, NAME => 'TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.', STARTKEY => '', ENDKEY => ''} 2023-04-21 01:54:17,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:17,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-21 01:54:17,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:17,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:17,132 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:17,134 DEBUG [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A 2023-04-21 01:54:17,134 DEBUG [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A 2023-04-21 01:54:17,134 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:54:17,134 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e00828c94a25954f7bdc6de3ac3e2662 columnFamilyName A 2023-04-21 01:54:17,135 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] regionserver.HStore(310): Store=e00828c94a25954f7bdc6de3ac3e2662/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:54:17,135 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:17,136 DEBUG [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B 2023-04-21 01:54:17,136 DEBUG [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B 2023-04-21 01:54:17,136 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:54:17,136 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e00828c94a25954f7bdc6de3ac3e2662 columnFamilyName B 2023-04-21 01:54:17,137 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] regionserver.HStore(310): Store=e00828c94a25954f7bdc6de3ac3e2662/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:54:17,137 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:17,138 DEBUG [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C 2023-04-21 01:54:17,138 DEBUG [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C 2023-04-21 01:54:17,138 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:54:17,139 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e00828c94a25954f7bdc6de3ac3e2662 columnFamilyName C 2023-04-21 01:54:17,139 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] regionserver.HStore(310): Store=e00828c94a25954f7bdc6de3ac3e2662/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:54:17,139 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:17,140 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:17,140 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:17,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-21 01:54:17,143 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:17,144 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-21 01:54:17,145 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e00828c94a25954f7bdc6de3ac3e2662; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71886755, jitterRate=0.07119612395763397}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-21 01:54:17,145 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:17,145 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., pid=18, masterSystemTime=1682042057126 2023-04-21 01:54:17,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:17,147 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:17,147 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=e00828c94a25954f7bdc6de3ac3e2662, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:17,147 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042057147"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682042057147"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682042057147"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682042057147"}]},"ts":"1682042057147"} 2023-04-21 01:54:17,150 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-21 01:54:17,151 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 in 174 msec 2023-04-21 01:54:17,153 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-21 01:54:17,153 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e00828c94a25954f7bdc6de3ac3e2662, ASSIGN in 332 msec 2023-04-21 01:54:17,153 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-21 01:54:17,154 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042057153"}]},"ts":"1682042057153"} 2023-04-21 01:54:17,155 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-21 01:54:17,158 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-21 01:54:17,160 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1710 sec 2023-04-21 01:54:18,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-21 01:54:18,096 INFO [Listener at localhost/39939] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-21 01:54:18,097 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x52274650 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:54:18,102 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7829b43e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:54:18,104 DEBUG [Listener at localhost/39939] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:54:18,108 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57816, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:54:18,110 DEBUG [Listener at localhost/39939] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-21 01:54:18,113 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34586, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-21 01:54:18,121 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-21 01:54:18,121 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-21 01:54:18,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-21 01:54:18,146 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-21 01:54:18,148 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-21 01:54:18,148 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-21 01:54:18,151 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-21 01:54:18,165 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e00828c94a25954f7bdc6de3ac3e2662, REOPEN/MOVE}] 2023-04-21 01:54:18,166 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e00828c94a25954f7bdc6de3ac3e2662, REOPEN/MOVE 2023-04-21 01:54:18,166 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=e00828c94a25954f7bdc6de3ac3e2662, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,166 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042058166"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682042058166"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682042058166"}]},"ts":"1682042058166"} 2023-04-21 01:54:18,168 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399}] 2023-04-21 01:54:18,320 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:18,321 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e00828c94a25954f7bdc6de3ac3e2662, disabling compactions & flushes 2023-04-21 01:54:18,321 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:18,321 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:18,321 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. after waiting 0 ms 2023-04-21 01:54:18,321 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:18,325 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-21 01:54:18,325 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:18,325 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:18,325 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: e00828c94a25954f7bdc6de3ac3e2662 to self. 2023-04-21 01:54:18,327 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:18,328 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=e00828c94a25954f7bdc6de3ac3e2662, regionState=CLOSED 2023-04-21 01:54:18,328 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042058327"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682042058327"}]},"ts":"1682042058327"} 2023-04-21 01:54:18,330 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-21 01:54:18,330 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 in 161 msec 2023-04-21 01:54:18,331 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e00828c94a25954f7bdc6de3ac3e2662, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,40131,1682042019399; forceNewPlan=false, retain=true 2023-04-21 01:54:18,481 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=e00828c94a25954f7bdc6de3ac3e2662, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,481 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042058481"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682042058481"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682042058481"}]},"ts":"1682042058481"} 2023-04-21 01:54:18,483 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399}] 2023-04-21 01:54:18,638 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:18,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e00828c94a25954f7bdc6de3ac3e2662, NAME => 'TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.', STARTKEY => '', ENDKEY => ''} 2023-04-21 01:54:18,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:18,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-21 01:54:18,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:18,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:18,641 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:18,642 DEBUG [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A 2023-04-21 01:54:18,642 DEBUG [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A 2023-04-21 01:54:18,642 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:54:18,645 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e00828c94a25954f7bdc6de3ac3e2662 columnFamilyName A 2023-04-21 01:54:18,646 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] regionserver.HStore(310): Store=e00828c94a25954f7bdc6de3ac3e2662/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:54:18,647 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:18,648 DEBUG [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B 2023-04-21 01:54:18,648 DEBUG [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B 2023-04-21 01:54:18,648 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:54:18,649 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e00828c94a25954f7bdc6de3ac3e2662 columnFamilyName B 2023-04-21 01:54:18,649 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] regionserver.HStore(310): Store=e00828c94a25954f7bdc6de3ac3e2662/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:54:18,649 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:18,650 DEBUG [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C 2023-04-21 01:54:18,650 DEBUG [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C 2023-04-21 01:54:18,650 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:54:18,650 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e00828c94a25954f7bdc6de3ac3e2662 columnFamilyName C 2023-04-21 01:54:18,651 INFO [StoreOpener-e00828c94a25954f7bdc6de3ac3e2662-1] regionserver.HStore(310): Store=e00828c94a25954f7bdc6de3ac3e2662/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:54:18,651 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:18,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:18,653 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:18,654 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-21 01:54:18,656 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:18,656 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e00828c94a25954f7bdc6de3ac3e2662; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71388813, jitterRate=0.06377620995044708}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-21 01:54:18,656 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:18,657 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., pid=23, masterSystemTime=1682042058635 2023-04-21 01:54:18,658 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:18,658 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:18,659 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=e00828c94a25954f7bdc6de3ac3e2662, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,659 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042058659"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682042058659"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682042058659"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682042058659"}]},"ts":"1682042058659"} 2023-04-21 01:54:18,661 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-21 01:54:18,661 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 in 177 msec 2023-04-21 01:54:18,663 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-21 01:54:18,663 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e00828c94a25954f7bdc6de3ac3e2662, REOPEN/MOVE in 496 msec 2023-04-21 01:54:18,665 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-21 01:54:18,665 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 514 msec 2023-04-21 01:54:18,668 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 541 msec 2023-04-21 01:54:18,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-21 01:54:18,677 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x34135972 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:54:18,681 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@33a47c05, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:54:18,682 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x02bab6b2 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:54:18,687 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1441c9f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:54:18,688 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x24bd3872 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:54:18,691 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@73e51571, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:54:18,692 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x370947cb to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:54:18,696 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6c1c92c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:54:18,697 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x210d1008 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:54:18,701 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c472cfd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:54:18,701 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x498694cb to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:54:18,706 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c4379c2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:54:18,707 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x421e5356 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:54:18,710 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f43b712, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:54:18,710 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x4a0fd7b4 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:54:18,713 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a3bb940, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:54:18,714 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x1a7b66a3 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:54:18,717 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2efdb02f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:54:18,725 DEBUG [hconnection-0x4c5cb840-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:54:18,725 DEBUG [hconnection-0x51c23be0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:54:18,726 DEBUG [hconnection-0x70417cb1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:54:18,726 DEBUG [hconnection-0x764abd48-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:54:18,726 DEBUG [hconnection-0x3c95254e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:54:18,727 DEBUG [hconnection-0x5e308efb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:54:18,727 DEBUG [hconnection-0x50dbd12f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:54:18,728 DEBUG [hconnection-0x5b0b6bef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:54:18,728 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57828, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:54:18,731 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:18,731 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57852, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:54:18,731 DEBUG [hconnection-0x4ff42959-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:54:18,731 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57830, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:54:18,731 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57848, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:54:18,733 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57888, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:54:18,738 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57864, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:54:18,739 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57890, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:54:18,751 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57878, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:54:18,752 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57904, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:54:18,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:18,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:18,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:18,761 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e00828c94a25954f7bdc6de3ac3e2662 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-21 01:54:18,761 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=A 2023-04-21 01:54:18,761 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:18,761 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=B 2023-04-21 01:54:18,761 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:18,761 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=C 2023-04-21 01:54:18,761 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:18,761 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:18,762 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:18,762 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:18,762 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:18,763 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:18,763 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:18,765 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,765 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:18,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:18,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:18,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:18,768 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,768 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:18,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:18,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:18,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:18,769 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:18,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:18,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:18,772 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:18,772 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:18,772 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:18,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682042058771, max timestamp=1682042058776Num uniques 859; 2023-04-21 01:54:18,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:54:18,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682042058771, max timestamp=1682042058776Num uniques 859; 2023-04-21 01:54:18,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682042058771, max timestamp=1682042058776Num uniques 859; 2023-04-21 01:54:18,780 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:54:18,780 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:54:18,788 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:18,788 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:18,788 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:18,788 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:18,788 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:18,788 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:18,788 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:18,788 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:18,788 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,794 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:18,794 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:18,795 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:18,796 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:18,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:18,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:18,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:18,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:18,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:18,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:18,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fcd3df2[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:18,798 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:18,798 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:18,798 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:18,799 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:18,799 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:18,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:18,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:18,800 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:18,800 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:18,800 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:18,800 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,800 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:18,802 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:18,802 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,802 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:18,802 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:18,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:18,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042118796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,805 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:18,805 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:18,805 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:18,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:18,805 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:18,805 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:18,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:18,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57852 deadline: 1682042118796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:18,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042118798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:18,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57878 deadline: 1682042118799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:18,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57864 deadline: 1682042118801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,808 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:18,808 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:18,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:18,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:18,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:18,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:18,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:18,809 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:18,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:18,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:18,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:18,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:18,811 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:18,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:18,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:18,812 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:18,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:18,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,815 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:18,815 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:18,815 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,816 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:18,816 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:18,816 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:18,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:18,816 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:18,816 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:18,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:18,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:18,816 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,816 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:18,816 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:18,816 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:18,816 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:18,816 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:18,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:18,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:18,844 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421983d5b7892b54975b97dd1ef43dad697 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421983d5b7892b54975b97dd1ef43dad697 2023-04-21 01:54:18,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:18,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042118907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:18,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57852 deadline: 1682042118907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:18,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042118908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:18,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57878 deadline: 1682042118910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:18,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57864 deadline: 1682042118910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,916 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:18,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:18,921 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:18,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:18,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:18,928 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:18,928 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:18,928 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:18,928 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:18,929 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:18,929 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:18,930 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:18,930 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:18,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:18,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:18,931 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,931 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:18,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:18,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:18,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:18,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:18,932 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:18,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:18,932 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:18,933 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:18,933 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:18,953 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:18,953 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:18,953 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:18,953 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:18,954 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:18,954 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:18,954 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:18,954 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:18,954 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,954 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:18,954 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:18,964 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,964 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:18,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:18,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:18,964 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:18,964 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:18,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:18,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:18,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:18,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:18,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:18,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66657068[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:18,966 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:18,966 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:18,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:18,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:18,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:18,967 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:18,967 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:18,967 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:18,968 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,968 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:18,970 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:18,970 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:18,970 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,970 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:18,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:18,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:18,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:18,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:18,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:18,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:18,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:18,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:18,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:18,973 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:18,973 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:18,973 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:18,973 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:18,973 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:18,975 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:18,975 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:18,975 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:18,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:18,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:18,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:18,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:18,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:18,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:18,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:18,976 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:18,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:18,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:18,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:18,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:18,978 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:18,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:18,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,983 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:18,983 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,983 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:18,983 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:18,983 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:18,983 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:18,983 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:18,983 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:18,983 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:18,983 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:18,983 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:18,983 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:18,987 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:18,987 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:18,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:18,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:19,084 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:19,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:19,088 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:19,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:19,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:19,092 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:19,092 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:19,092 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:19,092 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:19,093 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,093 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,094 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:19,094 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:19,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:19,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,095 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,095 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:19,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:19,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:19,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:19,100 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:19,100 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:19,100 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:19,104 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:19,104 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:19,105 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:19,105 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:19,105 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:19,105 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:19,105 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:19,105 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:19,105 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,105 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:19,105 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:19,107 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:19,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:19,107 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:19,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:19,108 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,108 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:19,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:19,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:19,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e36acb2[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:19,109 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:19,109 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,110 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,111 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:19,111 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:19,111 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:19,113 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,113 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:19,113 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:19,113 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:19,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:19,114 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:19,114 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:19,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042119111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,114 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:19,114 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:19,114 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:19,114 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:19,114 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:19,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:19,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042119112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,114 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:19,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57852 deadline: 1682042119111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:19,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57878 deadline: 1682042119115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,117 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:19,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57864 deadline: 1682042119117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,118 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,118 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:19,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,118 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:19,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:19,119 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:19,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,121 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,124 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:19,124 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:19,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:19,125 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,125 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,125 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,125 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,125 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:19,125 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,125 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:19,125 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:19,125 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:19,125 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,125 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:19,126 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:19,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:19,226 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:19,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:19,228 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:19,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:19,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:19,232 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:19,232 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:19,232 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:19,232 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:19,233 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,233 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,234 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,234 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:19,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:19,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:19,234 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,234 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:19,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:19,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,235 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:19,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:19,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:19,240 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:19,240 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:19,240 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:19,248 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:19,248 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:19,248 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:19,248 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:19,248 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:19,248 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:19,248 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:19,248 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:19,248 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,249 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:19,249 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:19,252 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,252 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,252 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:19,254 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:19,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:19,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@249c4765[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:19,258 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:19,258 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,259 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,259 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:19,259 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:19,259 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:19,259 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,260 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:19,261 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:19,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:19,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:19,262 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:19,262 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:19,262 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:19,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:19,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:19,264 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:19,264 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:19,264 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:19,264 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:19,264 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:19,266 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,266 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:19,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:19,267 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:19,268 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,269 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,272 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:19,272 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:19,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:19,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:19,273 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,273 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,273 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,273 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,273 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,273 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,273 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:19,273 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:19,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:19,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:19,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:19,273 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,273 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:19,274 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:19,274 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:19,280 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/fbc0f8c668444f9db87be37ec8ff75da 2023-04-21 01:54:19,302 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/18a02e8bc0b344f9bca5a5febbfc65da 2023-04-21 01:54:19,374 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:19,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:19,377 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:19,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:19,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:19,383 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:19,383 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:19,383 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:19,383 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:19,384 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,384 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,385 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:19,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:19,385 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,385 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,385 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:19,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:19,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:19,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:19,387 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:19,387 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:19,387 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:19,388 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:19,388 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:19,388 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:19,388 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:19,388 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:19,388 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:19,388 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:19,388 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:19,388 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,388 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:19,388 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:19,390 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,390 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:19,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:19,390 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,390 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:19,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:19,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:19,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:19,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:19,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4258cca9[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:19,393 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:19,393 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,395 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:19,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:19,395 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,395 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:19,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:19,400 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:19,400 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,400 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:19,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:19,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:19,401 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:19,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:19,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:19,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:19,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:19,404 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:19,404 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:19,404 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:19,404 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:19,404 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:19,405 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:19,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:19,406 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,406 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:19,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:19,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:19,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:19,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,407 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:19,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,408 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:19,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,411 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:19,411 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:19,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:19,411 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:19,411 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:19,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:19,411 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:19,411 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,411 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,411 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,411 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,411 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:19,411 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,411 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:19,411 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,411 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:19,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:19,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:19,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042119415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:19,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042119416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:19,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57852 deadline: 1682042119418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:19,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57864 deadline: 1682042119419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:19,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57878 deadline: 1682042119419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,511 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:19,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:19,514 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:19,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:19,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:19,516 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:19,516 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:19,516 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:19,516 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:19,516 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,516 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,518 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,518 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:19,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:19,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:19,518 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,518 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:19,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:19,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:19,519 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:19,519 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:19,519 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:19,519 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:19,519 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:19,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:19,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:19,520 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:19,520 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:19,520 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:19,520 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:19,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:19,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:19,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:19,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:19,523 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,523 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:19,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:19,523 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:19,523 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:19,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:19,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:19,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:19,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f7302ba[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:19,525 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:19,525 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,526 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,526 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:19,526 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:19,526 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,526 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:19,526 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:19,528 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,528 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:19,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:19,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:19,528 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:19,529 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:19,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:19,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:19,531 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:19,531 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:19,531 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:19,531 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:19,531 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:19,533 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,533 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:19,533 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:19,533 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,535 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,538 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,538 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,538 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,538 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,538 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,538 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:19,538 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,538 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,538 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:19,538 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:19,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:19,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:19,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:19,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:19,540 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:19,540 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:19,540 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:19,540 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:19,640 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:19,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:19,643 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:19,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:19,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:19,645 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:19,645 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:19,645 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:19,645 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:19,645 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,646 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,647 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:19,647 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:19,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:19,647 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,647 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:19,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:19,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:19,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:19,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:19,649 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:19,649 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:19,649 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:19,649 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:19,649 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:19,649 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:19,649 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:19,649 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:19,649 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:19,649 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:19,649 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:19,649 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,649 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:19,649 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:19,651 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,651 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:19,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:19,651 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,651 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:19,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:19,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:19,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:19,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:19,653 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:19,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@547b6cda[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:19,653 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,654 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,654 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:19,654 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:19,654 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:19,654 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,654 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:19,656 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,656 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:19,657 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:19,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:19,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:19,657 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:19,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:19,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:19,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:19,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:19,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:19,659 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:19,659 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:19,659 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:19,659 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:19,659 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:19,661 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,661 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,661 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:19,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:19,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:19,661 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:19,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:19,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:19,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:19,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,664 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:19,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:19,666 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:19,666 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:19,666 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:19,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:19,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:19,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:19,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:19,667 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:19,667 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:19,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:19,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:19,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:19,722 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/5091907d883d4b8fae3fd819660f3ef9 2023-04-21 01:54:19,728 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/fbc0f8c668444f9db87be37ec8ff75da as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/fbc0f8c668444f9db87be37ec8ff75da 2023-04-21 01:54:19,733 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/fbc0f8c668444f9db87be37ec8ff75da, entries=100, sequenceid=15, filesize=18.4 K 2023-04-21 01:54:19,734 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/18a02e8bc0b344f9bca5a5febbfc65da as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/18a02e8bc0b344f9bca5a5febbfc65da 2023-04-21 01:54:19,748 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/18a02e8bc0b344f9bca5a5febbfc65da, entries=100, sequenceid=15, filesize=9.3 K 2023-04-21 01:54:19,749 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/5091907d883d4b8fae3fd819660f3ef9 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/5091907d883d4b8fae3fd819660f3ef9 2023-04-21 01:54:19,754 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/5091907d883d4b8fae3fd819660f3ef9, entries=100, sequenceid=15, filesize=9.3 K 2023-04-21 01:54:19,755 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e00828c94a25954f7bdc6de3ac3e2662 in 994ms, sequenceid=15, compaction requested=false 2023-04-21 01:54:19,755 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:19,767 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:19,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:19,770 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:19,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:19,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:19,771 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:19,771 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:19,771 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:19,771 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:19,772 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,772 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,773 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,773 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:19,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:19,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:19,774 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,774 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:19,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:19,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:19,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:19,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:19,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:19,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:19,777 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:19,777 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:19,777 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:19,788 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:19,788 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:19,788 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:19,788 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:19,788 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e00828c94a25954f7bdc6de3ac3e2662 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-21 01:54:19,788 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=A 2023-04-21 01:54:19,788 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:19,789 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=B 2023-04-21 01:54:19,789 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:19,789 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=C 2023-04-21 01:54:19,789 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:19,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:19,920 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:19,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:19,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57878 deadline: 1682042119926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,930 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:19,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042119927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:19,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:19,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57864 deadline: 1682042119929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57852 deadline: 1682042119930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:19,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:19,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042119930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:20,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042120032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:20,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:20,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57852 deadline: 1682042120034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042120034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:20,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57864 deadline: 1682042120035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,224 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421b2cd07e5625247b6ba9a6a921b2ea1f6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421b2cd07e5625247b6ba9a6a921b2ea1f6 2023-04-21 01:54:20,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:20,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042120236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:20,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57852 deadline: 1682042120239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:20,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57864 deadline: 1682042120239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,241 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:20,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042120240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:20,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042120538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:20,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042120543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:20,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57852 deadline: 1682042120543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:20,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57864 deadline: 1682042120543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,634 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/7e00c2cebf3d4c8189a76c392a6c484c 2023-04-21 01:54:20,650 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/c13f074985214d19b03b3e475896cdbf 2023-04-21 01:54:20,669 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/8ae4f724d45b40e8bbca828c18e4a014 2023-04-21 01:54:20,674 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/7e00c2cebf3d4c8189a76c392a6c484c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/7e00c2cebf3d4c8189a76c392a6c484c 2023-04-21 01:54:20,678 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/7e00c2cebf3d4c8189a76c392a6c484c, entries=150, sequenceid=50, filesize=25.1 K 2023-04-21 01:54:20,679 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/c13f074985214d19b03b3e475896cdbf as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/c13f074985214d19b03b3e475896cdbf 2023-04-21 01:54:20,695 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/c13f074985214d19b03b3e475896cdbf, entries=150, sequenceid=50, filesize=11.6 K 2023-04-21 01:54:20,696 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/8ae4f724d45b40e8bbca828c18e4a014 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/8ae4f724d45b40e8bbca828c18e4a014 2023-04-21 01:54:20,702 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/8ae4f724d45b40e8bbca828c18e4a014, entries=150, sequenceid=50, filesize=11.6 K 2023-04-21 01:54:20,703 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for e00828c94a25954f7bdc6de3ac3e2662 in 915ms, sequenceid=50, compaction requested=false 2023-04-21 01:54:20,703 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:20,703 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:20,703 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:20,703 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:20,703 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,703 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:20,703 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:20,706 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,706 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:20,706 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:20,706 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:20,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:20,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@121aae5f[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:20,708 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:20,708 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:20,709 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:20,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:20,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:20,709 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,710 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:20,710 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:20,710 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:20,710 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:20,711 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:20,711 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:20,711 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:20,711 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:20,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:20,715 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:20,715 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:20,715 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:20,715 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:20,715 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:20,721 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:20,721 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:20,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:20,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:20,721 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:20,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:20,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:20,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:20,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:20,722 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:20,724 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:20,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:20,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,729 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,729 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:20,729 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:20,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:20,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:20,729 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:20,729 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:20,729 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:20,729 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,729 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:20,729 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:20,729 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:20,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:20,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:20,730 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:20,730 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:20,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:20,729 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:20,730 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:20,830 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:20,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:20,834 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:20,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:20,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:20,835 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:20,835 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:20,835 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:20,835 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:20,836 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:20,836 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:20,837 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:20,837 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:20,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:20,838 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:20,838 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:20,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:20,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:20,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:20,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:20,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:20,839 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:20,839 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:20,839 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:20,840 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:20,840 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:20,840 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:20,840 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:20,840 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e00828c94a25954f7bdc6de3ac3e2662 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-21 01:54:20,840 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=A 2023-04-21 01:54:20,841 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:20,841 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=B 2023-04-21 01:54:20,841 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:20,841 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=C 2023-04-21 01:54:20,841 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:20,884 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421c426ccd42e074caba4001e87120f392e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421c426ccd42e074caba4001e87120f392e 2023-04-21 01:54:20,900 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=61, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/4a55a36917f14340a7700ed09daa7987 2023-04-21 01:54:20,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:20,956 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:21,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682042060994, max timestamp=1682042061008Num uniques 859; 2023-04-21 01:54:21,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682042060994, max timestamp=1682042061008Num uniques 859; 2023-04-21 01:54:21,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:54:21,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682042060994, max timestamp=1682042061008Num uniques 859; 2023-04-21 01:54:21,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:54:21,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:54:21,025 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-21 01:54:21,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:21,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57852 deadline: 1682042121047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:21,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57864 deadline: 1682042121047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:21,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57878 deadline: 1682042121051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:21,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042121051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:21,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042121052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:21,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57878 deadline: 1682042121155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,159 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:21,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042121159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:21,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042121159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,320 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/2f52a25ed41141a09da755973c49d355 2023-04-21 01:54:21,340 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/c6b32ede87ce4ee4bcf7ae7dcd7bf3bd 2023-04-21 01:54:21,348 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/4a55a36917f14340a7700ed09daa7987 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/4a55a36917f14340a7700ed09daa7987 2023-04-21 01:54:21,354 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/4a55a36917f14340a7700ed09daa7987, entries=100, sequenceid=61, filesize=18.4 K 2023-04-21 01:54:21,355 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/2f52a25ed41141a09da755973c49d355 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/2f52a25ed41141a09da755973c49d355 2023-04-21 01:54:21,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:21,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57878 deadline: 1682042121359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:21,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042121361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:21,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042121362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,364 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/2f52a25ed41141a09da755973c49d355, entries=100, sequenceid=61, filesize=9.3 K 2023-04-21 01:54:21,366 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/c6b32ede87ce4ee4bcf7ae7dcd7bf3bd as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/c6b32ede87ce4ee4bcf7ae7dcd7bf3bd 2023-04-21 01:54:21,371 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/c6b32ede87ce4ee4bcf7ae7dcd7bf3bd, entries=100, sequenceid=61, filesize=9.3 K 2023-04-21 01:54:21,371 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e00828c94a25954f7bdc6de3ac3e2662 in 531ms, sequenceid=61, compaction requested=true 2023-04-21 01:54:21,372 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:21,372 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:21,372 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:21,372 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:21,372 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,372 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:21,372 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:21,374 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:21,374 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:21,374 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:21,374 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:21,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:21,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16230531[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:21,376 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:21,376 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:21,379 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:21,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:21,379 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:21,379 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:21,379 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:21,379 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,379 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:21,381 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:21,381 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:21,381 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:21,381 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:21,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:21,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:21,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:21,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:21,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:21,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:21,384 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:21,384 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:21,384 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:21,384 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:21,384 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:21,387 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:21,387 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:21,387 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:21,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:21,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:21,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:21,388 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:21,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,390 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:21,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,392 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,392 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:21,392 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:21,392 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,392 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:21,392 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:21,392 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:21,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:21,392 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:21,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:21,392 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:21,392 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:21,392 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:21,392 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:21,392 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:21,393 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:21,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:21,493 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:21,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:21,495 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:21,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:21,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:21,496 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:21,496 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:21,496 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:21,496 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:21,497 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:21,497 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:21,499 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:21,499 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:21,500 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,500 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:21,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:21,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:21,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:21,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:21,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:21,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:21,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:21,501 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:21,501 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:21,501 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:21,501 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:21,501 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:21,501 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:21,501 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:21,502 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e00828c94a25954f7bdc6de3ac3e2662 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-21 01:54:21,502 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=A 2023-04-21 01:54:21,502 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:21,502 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=B 2023-04-21 01:54:21,502 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:21,502 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=C 2023-04-21 01:54:21,502 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:21,519 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304216dfce49ec0ae459d8687e2998e57f24c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304216dfce49ec0ae459d8687e2998e57f24c 2023-04-21 01:54:21,525 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=96, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/30594e1a61754b6ea41c587a59d5d7ec 2023-04-21 01:54:21,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:21,665 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:21,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:21,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042121670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:21,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57878 deadline: 1682042121671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:21,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042121674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:21,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042121776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:21,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57878 deadline: 1682042121777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:21,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042121777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,952 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/d12924fab43644d1967d453255d5a488 2023-04-21 01:54:21,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:21,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042121979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:21,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57878 deadline: 1682042121980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:21,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:21,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042121981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:22,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57864 deadline: 1682042122060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:22,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57852 deadline: 1682042122060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:22,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042122282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:22,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57878 deadline: 1682042122282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:22,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042122285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,370 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/00ab17fdc3994fec9f7a6ea69ac6f6cd 2023-04-21 01:54:22,375 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/30594e1a61754b6ea41c587a59d5d7ec as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/30594e1a61754b6ea41c587a59d5d7ec 2023-04-21 01:54:22,379 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/30594e1a61754b6ea41c587a59d5d7ec, entries=150, sequenceid=96, filesize=25.1 K 2023-04-21 01:54:22,380 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/d12924fab43644d1967d453255d5a488 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/d12924fab43644d1967d453255d5a488 2023-04-21 01:54:22,387 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/d12924fab43644d1967d453255d5a488, entries=150, sequenceid=96, filesize=11.6 K 2023-04-21 01:54:22,388 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/00ab17fdc3994fec9f7a6ea69ac6f6cd as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/00ab17fdc3994fec9f7a6ea69ac6f6cd 2023-04-21 01:54:22,393 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/00ab17fdc3994fec9f7a6ea69ac6f6cd, entries=150, sequenceid=96, filesize=11.6 K 2023-04-21 01:54:22,394 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for e00828c94a25954f7bdc6de3ac3e2662 in 892ms, sequenceid=96, compaction requested=true 2023-04-21 01:54:22,395 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:22,395 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:22,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:22,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:22,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:22,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:22,396 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:22,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:22,396 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:22,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:22,397 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:22,397 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:22,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:22,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:22,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:22,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:22,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d2a39d[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:22,398 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:22,399 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:22,400 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:22,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:22,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:22,400 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:22,400 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:22,400 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:22,400 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,400 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:22,401 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,401 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:22,401 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:22,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:22,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:22,401 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:22,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:22,404 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:22,404 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:22,404 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:22,404 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:22,404 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:22,407 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:22,407 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:22,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:22,407 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:22,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:22,407 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:22,409 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,411 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,411 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:22,411 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:22,412 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:22,412 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,412 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:22,412 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:22,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:22,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:22,412 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:22,412 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:22,412 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:22,412 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:22,412 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:22,412 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:22,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:22,412 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:22,512 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:22,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:22,515 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:22,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:22,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:22,516 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:22,516 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:22,516 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:22,516 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:22,517 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:22,517 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:22,518 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:22,518 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:22,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:22,518 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,518 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:22,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:22,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:22,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:22,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:22,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:22,519 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:22,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:22,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:22,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:22,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:22,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:22,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:22,520 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:22,520 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:22,520 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e00828c94a25954f7bdc6de3ac3e2662 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-21 01:54:22,520 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=A 2023-04-21 01:54:22,521 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:22,521 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=B 2023-04-21 01:54:22,521 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:22,521 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=C 2023-04-21 01:54:22,521 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:22,542 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304212658eea2de5a442799e88b2fc15877d4 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304212658eea2de5a442799e88b2fc15877d4 2023-04-21 01:54:22,562 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=107, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/bed55f6840e64da59359cea49d668e5b 2023-04-21 01:54:22,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:22,786 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:22,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682042062792, max timestamp=1682042062808Num uniques 859; 2023-04-21 01:54:22,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682042062792, max timestamp=1682042062808Num uniques 859; 2023-04-21 01:54:22,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682042062792, max timestamp=1682042062808Num uniques 859; 2023-04-21 01:54:22,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:22,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-21 01:54:22,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042122807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-21 01:54:22,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-21 01:54:22,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:22,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042122829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:22,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57878 deadline: 1682042122829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:22,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042122911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:22,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:22,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042122933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57878 deadline: 1682042122933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:22,990 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/39a50432a99647c2b1a74b8ffa73623f 2023-04-21 01:54:23,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:23,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042123113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:23,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57878 deadline: 1682042123138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:23,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042123138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,402 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/8298aa1a5f0247b3b7b6e012b5b04219 2023-04-21 01:54:23,407 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/bed55f6840e64da59359cea49d668e5b as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/bed55f6840e64da59359cea49d668e5b 2023-04-21 01:54:23,411 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/bed55f6840e64da59359cea49d668e5b, entries=150, sequenceid=107, filesize=25.1 K 2023-04-21 01:54:23,412 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/39a50432a99647c2b1a74b8ffa73623f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/39a50432a99647c2b1a74b8ffa73623f 2023-04-21 01:54:23,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:23,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042123416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,418 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/39a50432a99647c2b1a74b8ffa73623f, entries=150, sequenceid=107, filesize=11.6 K 2023-04-21 01:54:23,419 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/8298aa1a5f0247b3b7b6e012b5b04219 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/8298aa1a5f0247b3b7b6e012b5b04219 2023-04-21 01:54:23,424 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/8298aa1a5f0247b3b7b6e012b5b04219, entries=150, sequenceid=107, filesize=11.6 K 2023-04-21 01:54:23,425 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for e00828c94a25954f7bdc6de3ac3e2662 in 905ms, sequenceid=107, compaction requested=true 2023-04-21 01:54:23,425 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:23,425 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:23,425 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:23,425 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:23,426 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,426 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:23,426 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:23,427 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,427 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:23,428 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:23,428 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:23,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:23,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:23,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:23,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e4002c1[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:23,429 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:23,429 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,431 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,431 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:23,431 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:23,431 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:23,431 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,431 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:23,432 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:23,432 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:23,432 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:23,433 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:23,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:23,436 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:23,436 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:23,437 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:23,437 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:23,437 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:23,439 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,439 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,439 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:23,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:23,440 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:23,441 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e00828c94a25954f7bdc6de3ac3e2662 3/3 column families, dataSize=140.89 KB heapSize=386.16 KB 2023-04-21 01:54:23,442 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=A 2023-04-21 01:54:23,442 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:23,442 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=B 2023-04-21 01:54:23,442 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,442 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=C 2023-04-21 01:54:23,442 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:23,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:23,443 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,447 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:23,447 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:23,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:23,447 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:23,447 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:23,447 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:23,448 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:23,448 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:23,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:23,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:23,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:23,447 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,452 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:23,452 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:23,452 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,452 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,452 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,453 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:23,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042123454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,462 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304218e054822abc8434297030a3b7f57c07c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304218e054822abc8434297030a3b7f57c07c 2023-04-21 01:54:23,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:23,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57878 deadline: 1682042123458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,548 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:23,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:23,550 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:23,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:23,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:23,552 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:23,552 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:23,552 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:23,552 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:23,553 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,553 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:23,554 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:23,554 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:23,554 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,554 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:23,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:23,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:23,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:23,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:23,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:23,556 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:23,556 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:23,556 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:23,557 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:23,557 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:23,557 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:23,557 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:23,557 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:23,557 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:23,557 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:23,557 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:23,557 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,557 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:23,557 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:23,559 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:23,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:23,560 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:23,560 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,560 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:23,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:23,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:23,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:23,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@496f2d00[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:23,562 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:23,562 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,563 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:23,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:23,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:23,563 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,563 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:23,564 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,564 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:23,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:23,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:23,564 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:23,565 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:23,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:23,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:23,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:23,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:23,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042123559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:23,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:23,567 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:23,567 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:23,567 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:23,567 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:23,567 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:23,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:23,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57878 deadline: 1682042123565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,569 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:23,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:23,569 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,569 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:23,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:23,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:23,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:23,570 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:23,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,572 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:23,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,576 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,576 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:23,576 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:23,576 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,576 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:23,576 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:23,576 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:23,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:23,576 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:23,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:23,576 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:23,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:23,576 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,577 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,577 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,577 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:23,577 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:23,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:23,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:23,677 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:23,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:23,679 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:23,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:23,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:23,681 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:23,681 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:23,681 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:23,681 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:23,681 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,681 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:23,683 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:23,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:23,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:23,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:23,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,684 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:23,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:23,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:23,684 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,684 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:23,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:23,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:23,687 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:23,687 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:23,687 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:23,687 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:23,687 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:23,687 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:23,687 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:23,687 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:23,687 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:23,687 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:23,687 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:23,687 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,687 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:23,687 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:23,688 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,688 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:23,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:23,689 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,689 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:23,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:23,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:23,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:23,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:23,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47ebb391[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:23,690 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:23,691 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,692 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,692 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:23,692 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:23,692 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:23,692 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,692 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:23,694 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:23,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:23,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:23,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:23,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:23,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:23,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:23,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:23,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:23,697 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:23,697 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:23,697 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:23,697 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:23,697 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:23,699 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,699 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:23,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,700 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:23,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:23,701 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:23,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,702 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,705 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:23,705 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,705 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:23,705 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:23,705 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:23,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:23,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:23,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:23,705 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:23,705 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:23,705 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:23,705 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:23,706 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:23,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:23,706 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:23,706 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,706 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,706 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,770 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:23,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042123769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,770 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:23,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57878 deadline: 1682042123769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,806 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:23,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:23,808 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:23,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:23,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:23,809 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:23,809 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:23,809 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:23,809 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:23,810 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,810 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:23,811 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,811 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:23,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:23,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:23,811 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,811 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:23,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:23,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:23,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:23,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:23,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:23,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:23,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:23,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:23,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:23,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:23,813 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:23,813 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:23,813 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:23,813 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:23,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:23,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:23,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:23,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:23,816 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,816 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:23,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:23,816 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,816 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:23,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:23,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:23,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:23,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:23,818 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:23,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53b13eba[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:23,818 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,819 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,819 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:23,819 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:23,819 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:23,819 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,819 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:23,820 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,820 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:23,820 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:23,820 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:23,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:23,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:23,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:23,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:23,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:23,823 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:23,823 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:23,823 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:23,823 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:23,823 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:23,825 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,825 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:23,825 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:23,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,826 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:23,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,827 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:23,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:23,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:23,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:23,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:23,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:23,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,835 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:23,835 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:23,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:23,835 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:23,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,835 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:23,835 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:23,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:23,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:23,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:23,870 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=141, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/06489ef226974324b8669898ec881ab0 2023-04-21 01:54:23,883 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/14268ab08d464ca49e7b2b021c9b8ec1 2023-04-21 01:54:23,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:23,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042123921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,935 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:23,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:23,938 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:23,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:23,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:23,939 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:23,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:23,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:23,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:23,940 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,940 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:23,941 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:23,941 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:23,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:23,941 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:23,941 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:23,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:23,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:23,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:23,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:23,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:23,942 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:23,942 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:23,942 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:23,943 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:23,943 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:23,943 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:23,943 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:23,943 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:23,943 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:23,943 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:23,943 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:23,943 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,943 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:23,943 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:23,944 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,944 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:23,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:23,945 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:23,945 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:23,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:23,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:23,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:23,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d697807[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:23,946 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:23,946 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,947 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,947 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:23,947 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:23,948 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:23,948 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,948 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:23,950 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:23,950 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,950 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:23,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:23,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:23,950 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:23,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:23,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:23,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:23,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:23,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:23,953 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:23,953 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:23,953 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:23,953 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:23,953 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:23,956 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,956 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,956 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:23,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:23,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:23,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:23,956 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:23,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:23,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,958 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,960 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,960 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:23,960 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:23,960 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:23,960 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:23,960 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:23,960 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,960 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:23,960 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:23,960 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:23,960 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:23,960 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:23,960 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:23,961 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:23,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:23,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:23,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:23,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:23,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:24,061 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:24,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:24,063 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:24,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:24,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:24,064 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:24,065 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:24,065 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:24,065 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:24,065 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,065 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:24,066 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:24,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:24,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:24,067 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,067 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:24,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:24,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:24,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:24,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:24,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:24,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:24,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:24,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:24,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:24,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:24,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:24,068 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:24,068 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:24,069 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:24,069 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:24,069 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:24,069 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:24,069 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,069 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:24,069 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:24,071 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:24,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:24,071 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:24,071 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,071 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:24,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:24,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:24,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:24,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dde5987[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:24,073 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:24,073 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:24,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042124071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:24,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57878 deadline: 1682042124072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,074 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,074 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:24,074 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:24,075 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:24,074 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,075 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:24,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:24,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57864 deadline: 1682042124074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,076 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,076 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:24,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:24,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:24,076 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:24,077 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:24,078 DEBUG [Thread-1014] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:54:24,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:24,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:24,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:24,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:24,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:24,080 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:24,080 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:24,080 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:24,080 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:24,080 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:24,082 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,082 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:24,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,082 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:24,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:24,083 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:24,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57852 deadline: 1682042124080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:24,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,084 DEBUG [Thread-1020] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:54:24,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:24,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,085 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:24,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,088 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,088 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:24,088 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:24,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:24,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:24,088 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:24,088 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:24,089 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,089 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:24,089 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:24,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:24,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:24,089 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:24,089 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,089 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,089 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:24,089 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:24,089 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,189 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:24,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:24,192 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:24,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:24,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:24,193 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:24,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:24,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:24,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:24,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:24,194 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:24,195 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:24,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:24,195 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,195 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:24,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:24,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:24,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:24,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:24,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:24,196 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:24,196 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:24,196 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:24,196 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:24,196 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:24,196 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:24,196 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:24,196 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:24,196 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:24,197 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:24,197 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:24,197 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,197 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:24,197 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:24,198 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,198 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:24,198 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,198 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:24,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:24,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:24,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:24,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77b30896[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:24,200 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:24,200 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,201 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:24,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:24,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:24,201 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,201 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:24,203 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,203 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:24,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:24,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:24,203 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:24,203 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:24,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:24,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:24,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:24,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:24,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:24,206 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:24,206 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:24,206 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:24,206 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:24,206 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:24,208 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,208 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,208 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:24,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:24,208 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:24,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,210 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:24,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,212 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,212 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:24,212 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:24,212 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:24,212 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:24,212 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,212 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,212 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,212 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,212 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:24,212 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:24,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:24,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:24,212 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:24,212 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:24,212 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:24,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:24,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:24,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:24,306 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/3adfb855ac3e4dd39653a2865359a96f 2023-04-21 01:54:24,312 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:24,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:24,317 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:24,317 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/06489ef226974324b8669898ec881ab0 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/06489ef226974324b8669898ec881ab0 2023-04-21 01:54:24,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:24,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:24,319 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:24,319 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:24,319 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:24,319 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:24,319 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,319 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:24,320 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:24,320 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:24,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:24,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:24,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:24,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:24,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:24,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:24,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:24,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:24,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:24,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:24,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:24,324 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/06489ef226974324b8669898ec881ab0, entries=150, sequenceid=141, filesize=25.3 K 2023-04-21 01:54:24,325 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/14268ab08d464ca49e7b2b021c9b8ec1 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/14268ab08d464ca49e7b2b021c9b8ec1 2023-04-21 01:54:24,332 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:24,332 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:24,332 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:24,332 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:24,332 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:24,332 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:24,332 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:24,332 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:24,332 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,332 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:24,332 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:24,336 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/14268ab08d464ca49e7b2b021c9b8ec1, entries=150, sequenceid=141, filesize=11.8 K 2023-04-21 01:54:24,337 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/3adfb855ac3e4dd39653a2865359a96f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/3adfb855ac3e4dd39653a2865359a96f 2023-04-21 01:54:24,338 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,338 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:24,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:24,338 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,338 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:24,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:24,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:24,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:24,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:24,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dadbc34[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:24,343 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:24,343 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,344 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,344 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:24,345 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:24,345 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:24,345 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/3adfb855ac3e4dd39653a2865359a96f, entries=150, sequenceid=141, filesize=11.8 K 2023-04-21 01:54:24,345 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,346 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:24,346 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~386.11 KB/395376, currentSize=46.96 KB/48090 for e00828c94a25954f7bdc6de3ac3e2662 in 905ms, sequenceid=141, compaction requested=true 2023-04-21 01:54:24,346 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:24,346 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:24,346 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:24,346 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-21 01:54:24,346 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-21 01:54:24,346 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:24,347 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:24,347 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:24,347 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:24,347 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:24,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:24,348 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 66778 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-21 01:54:24,348 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): e00828c94a25954f7bdc6de3ac3e2662/B is initiating minor compaction (all files) 2023-04-21 01:54:24,348 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e00828c94a25954f7bdc6de3ac3e2662/B in TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:24,348 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/18a02e8bc0b344f9bca5a5febbfc65da, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/c13f074985214d19b03b3e475896cdbf, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/2f52a25ed41141a09da755973c49d355, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/d12924fab43644d1967d453255d5a488, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/39a50432a99647c2b1a74b8ffa73623f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/14268ab08d464ca49e7b2b021c9b8ec1] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp, totalSize=65.2 K 2023-04-21 01:54:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,349 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 18a02e8bc0b344f9bca5a5febbfc65da, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682042058756 2023-04-21 01:54:24,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,349 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 140730 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-21 01:54:24,349 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): e00828c94a25954f7bdc6de3ac3e2662/A is initiating minor compaction (all files) 2023-04-21 01:54:24,349 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting c13f074985214d19b03b3e475896cdbf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682042058794 2023-04-21 01:54:24,349 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e00828c94a25954f7bdc6de3ac3e2662/A in TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:24,349 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/fbc0f8c668444f9db87be37ec8ff75da, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/7e00c2cebf3d4c8189a76c392a6c484c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/4a55a36917f14340a7700ed09daa7987, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/30594e1a61754b6ea41c587a59d5d7ec, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/bed55f6840e64da59359cea49d668e5b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/06489ef226974324b8669898ec881ab0] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp, totalSize=137.4 K 2023-04-21 01:54:24,350 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 2f52a25ed41141a09da755973c49d355, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1682042059928 2023-04-21 01:54:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:24,350 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting fbc0f8c668444f9db87be37ec8ff75da, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682042058756 2023-04-21 01:54:24,350 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting d12924fab43644d1967d453255d5a488, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1682042061036 2023-04-21 01:54:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,350 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 7e00c2cebf3d4c8189a76c392a6c484c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682042058794 2023-04-21 01:54:24,351 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 39a50432a99647c2b1a74b8ffa73623f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1682042061667 2023-04-21 01:54:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,351 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 4a55a36917f14340a7700ed09daa7987, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1682042059928 2023-04-21 01:54:24,351 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 14268ab08d464ca49e7b2b021c9b8ec1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1682042062824 2023-04-21 01:54:24,352 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 30594e1a61754b6ea41c587a59d5d7ec, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1682042061036 2023-04-21 01:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:24,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:24,352 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:24,352 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:24,352 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:24,352 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:24,352 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:24,352 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting bed55f6840e64da59359cea49d668e5b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1682042061667 2023-04-21 01:54:24,353 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 06489ef226974324b8669898ec881ab0, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1682042062824 2023-04-21 01:54:24,354 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,354 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:24,354 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:24,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:24,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:24,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,355 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:24,357 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,361 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,361 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:24,361 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:24,361 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,361 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,361 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:24,361 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:24,362 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:24,361 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:24,361 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,362 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:24,362 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:24,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:24,362 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:24,362 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:24,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:24,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:24,375 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): e00828c94a25954f7bdc6de3ac3e2662#B#compaction#160 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:24,392 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): e00828c94a25954f7bdc6de3ac3e2662#A#compaction#161 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:24,397 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/9d2b87195832418bb6c706bc49348be5 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/9d2b87195832418bb6c706bc49348be5 2023-04-21 01:54:24,404 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e00828c94a25954f7bdc6de3ac3e2662/B of e00828c94a25954f7bdc6de3ac3e2662 into 9d2b87195832418bb6c706bc49348be5(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:24,404 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:24,404 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., storeName=e00828c94a25954f7bdc6de3ac3e2662/B, priority=10, startTime=1682042064346; duration=0sec 2023-04-21 01:54:24,404 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:24,404 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-21 01:54:24,406 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 66778 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-21 01:54:24,406 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): e00828c94a25954f7bdc6de3ac3e2662/C is initiating minor compaction (all files) 2023-04-21 01:54:24,407 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e00828c94a25954f7bdc6de3ac3e2662/C in TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:24,407 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/5091907d883d4b8fae3fd819660f3ef9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/8ae4f724d45b40e8bbca828c18e4a014, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/c6b32ede87ce4ee4bcf7ae7dcd7bf3bd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/00ab17fdc3994fec9f7a6ea69ac6f6cd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/8298aa1a5f0247b3b7b6e012b5b04219, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/3adfb855ac3e4dd39653a2865359a96f] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp, totalSize=65.2 K 2023-04-21 01:54:24,407 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 5091907d883d4b8fae3fd819660f3ef9, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682042058756 2023-04-21 01:54:24,408 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 8ae4f724d45b40e8bbca828c18e4a014, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682042058794 2023-04-21 01:54:24,408 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting c6b32ede87ce4ee4bcf7ae7dcd7bf3bd, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1682042059928 2023-04-21 01:54:24,408 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 00ab17fdc3994fec9f7a6ea69ac6f6cd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1682042061036 2023-04-21 01:54:24,409 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 8298aa1a5f0247b3b7b6e012b5b04219, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1682042061667 2023-04-21 01:54:24,410 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 3adfb855ac3e4dd39653a2865359a96f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1682042062824 2023-04-21 01:54:24,436 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): e00828c94a25954f7bdc6de3ac3e2662#C#compaction#162 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:24,453 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/296a14adbbff4ebbb7e25742b414e2ec as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/296a14adbbff4ebbb7e25742b414e2ec 2023-04-21 01:54:24,459 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e00828c94a25954f7bdc6de3ac3e2662/C of e00828c94a25954f7bdc6de3ac3e2662 into 296a14adbbff4ebbb7e25742b414e2ec(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:24,459 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:24,459 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., storeName=e00828c94a25954f7bdc6de3ac3e2662/C, priority=10, startTime=1682042064346; duration=0sec 2023-04-21 01:54:24,459 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:24,462 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:24,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:24,465 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:24,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:24,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:24,466 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:24,466 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:24,466 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:24,466 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:24,467 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,467 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:24,468 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:24,468 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:24,469 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,469 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:24,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:24,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:24,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:24,470 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:24,470 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:24,470 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:24,470 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:24,470 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:24,472 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:24,472 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:24,472 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:24,472 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:24,472 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e00828c94a25954f7bdc6de3ac3e2662 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-21 01:54:24,473 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=A 2023-04-21 01:54:24,473 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:24,473 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=B 2023-04-21 01:54:24,473 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:24,473 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=C 2023-04-21 01:54:24,473 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:24,499 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421986abb4c4f8a4779bdc0b3d3d5ba767e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421986abb4c4f8a4779bdc0b3d3d5ba767e 2023-04-21 01:54:24,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:24,578 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:24,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042064632, max timestamp=1682042064635Num uniques 859; 2023-04-21 01:54:24,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042064632, max timestamp=1682042064635Num uniques 859; 2023-04-21 01:54:24,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:54:24,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042064632, max timestamp=1682042064635Num uniques 859; 2023-04-21 01:54:24,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:54:24,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:54:24,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:24,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042124666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:24,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57878 deadline: 1682042124670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:24,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57888 deadline: 1682042124771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:24,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57878 deadline: 1682042124771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,867 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/2b5f7650ffa942d083f5b4ecef724614 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/2b5f7650ffa942d083f5b4ecef724614 2023-04-21 01:54:24,873 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e00828c94a25954f7bdc6de3ac3e2662/A of e00828c94a25954f7bdc6de3ac3e2662 into 2b5f7650ffa942d083f5b4ecef724614(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:24,873 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:24,873 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., storeName=e00828c94a25954f7bdc6de3ac3e2662/A, priority=10, startTime=1682042064346; duration=0sec 2023-04-21 01:54:24,873 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:24,906 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=153, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/044e2cb029584348b6b441caf0adf245 2023-04-21 01:54:24,918 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/1a5fca224cb049a683942fe74af11f77 2023-04-21 01:54:24,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:24,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042124929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,934 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/98f4b7fcbb2245718e7454ab0747b10a 2023-04-21 01:54:24,939 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/044e2cb029584348b6b441caf0adf245 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/044e2cb029584348b6b441caf0adf245 2023-04-21 01:54:24,945 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/044e2cb029584348b6b441caf0adf245, entries=100, sequenceid=153, filesize=18.5 K 2023-04-21 01:54:24,946 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/1a5fca224cb049a683942fe74af11f77 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/1a5fca224cb049a683942fe74af11f77 2023-04-21 01:54:24,952 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/1a5fca224cb049a683942fe74af11f77, entries=100, sequenceid=153, filesize=9.4 K 2023-04-21 01:54:24,953 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/98f4b7fcbb2245718e7454ab0747b10a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/98f4b7fcbb2245718e7454ab0747b10a 2023-04-21 01:54:24,958 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/98f4b7fcbb2245718e7454ab0747b10a, entries=100, sequenceid=153, filesize=9.4 K 2023-04-21 01:54:24,959 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e00828c94a25954f7bdc6de3ac3e2662 in 487ms, sequenceid=153, compaction requested=false 2023-04-21 01:54:24,959 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:24,959 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:24,959 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:24,959 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:24,959 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,959 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:24,959 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:24,961 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,961 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:24,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:24,962 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,962 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:24,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:24,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:24,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:24,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b1240c6[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:24,964 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:24,964 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,965 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,965 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:24,966 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:24,966 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:24,967 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,967 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:24,969 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,969 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:24,969 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:24,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:24,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:24,969 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:24,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:24,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:24,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:24,973 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:24,973 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:24,973 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:24,973 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:24,973 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:24,975 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,975 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:24,976 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:24,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:24,977 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:24,978 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e00828c94a25954f7bdc6de3ac3e2662 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-21 01:54:24,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682042064649, max timestamp=1682042064976Num uniques 1018; 2023-04-21 01:54:24,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-21 01:54:24,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682042064649, max timestamp=1682042064976Num uniques 1018; 2023-04-21 01:54:24,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-21 01:54:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,979 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682042064649, max timestamp=1682042064976Num uniques 1018; 2023-04-21 01:54:24,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-21 01:54:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:24,981 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=A 2023-04-21 01:54:24,981 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:24,981 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=B 2023-04-21 01:54:24,981 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:24,981 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=C 2023-04-21 01:54:24,981 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,983 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,983 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:24,983 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:24,983 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:24,983 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:24,983 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:24,983 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:24,983 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:24,983 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,983 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:24,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:24,983 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:24,983 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:24,983 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:24,983 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:24,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:24,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:25,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042065002, max timestamp=1682042065009Num uniques 859; 2023-04-21 01:54:25,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:54:25,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042065002, max timestamp=1682042065009Num uniques 859; 2023-04-21 01:54:25,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042065002, max timestamp=1682042065009Num uniques 859; 2023-04-21 01:54:25,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:54:25,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:54:25,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682042065034, max timestamp=1682042065043Num uniques 1018; 2023-04-21 01:54:25,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-21 01:54:25,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682042065034, max timestamp=1682042065043Num uniques 1018; 2023-04-21 01:54:25,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-21 01:54:25,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682042065034, max timestamp=1682042065043Num uniques 1018; 2023-04-21 01:54:25,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-21 01:54:25,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682042065060, max timestamp=1682042065071Num uniques 1027; 2023-04-21 01:54:25,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-21 01:54:25,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682042065060, max timestamp=1682042065071Num uniques 1027; 2023-04-21 01:54:25,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682042065060, max timestamp=1682042065071Num uniques 1027; 2023-04-21 01:54:25,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-21 01:54:25,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-21 01:54:25,084 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:25,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:25,086 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:25,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:25,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:25,087 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:25,087 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:25,087 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:25,087 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:25,088 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,088 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,089 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:25,089 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:25,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:25,089 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,089 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:25,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:25,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,090 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:25,090 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:25,090 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:25,090 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:25,090 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:25,091 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:25,091 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:25,091 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:25,091 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:25,091 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:25,091 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:25,091 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:25,091 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:25,091 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:25,091 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,091 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:25,091 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:25,094 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,094 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:25,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:25,094 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,094 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:25,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:25,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:25,096 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:25,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@241f2442[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:25,096 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,097 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,097 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:25,097 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:25,097 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,097 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:25,097 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:25,099 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,099 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:25,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:25,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:25,099 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:25,099 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:25,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:25,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:25,101 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:25,101 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:25,102 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:25,102 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:25,102 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:25,103 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,103 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,103 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:25,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:25,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:25,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:25,104 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,105 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:25,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1682042065081, max timestamp=1682042065105Num uniques 1086; 2023-04-21 01:54:25,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=15386, suffix heap size=140912,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=336 2023-04-21 01:54:25,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1682042065081, max timestamp=1682042065105Num uniques 1086; 2023-04-21 01:54:25,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=15386, suffix heap size=140912,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=336 2023-04-21 01:54:25,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1682042065081, max timestamp=1682042065105Num uniques 1086; 2023-04-21 01:54:25,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=15386, suffix heap size=140912,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=336 2023-04-21 01:54:25,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,112 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,112 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:25,112 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,112 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:25,112 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:25,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:25,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:25,112 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:25,112 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:25,112 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,112 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,112 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,112 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,112 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,112 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:25,112 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:25,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:25,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:25,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57878 deadline: 1682042125135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:25,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57888 deadline: 1682042125136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,212 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:25,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:25,215 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:25,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:25,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:25,216 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:25,216 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:25,216 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:25,216 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:25,217 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,217 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,218 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:25,218 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:25,218 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,218 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:25,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:25,219 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:25,219 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:25,219 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:25,219 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:25,220 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:25,220 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:25,220 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:25,220 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:25,220 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:25,220 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:25,220 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:25,220 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:25,220 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:25,220 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,220 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:25,220 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:25,222 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:25,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:25,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b877fef[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:25,223 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:25,224 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,225 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,225 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:25,225 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:25,225 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:25,225 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,225 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:25,230 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,230 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:25,230 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:25,230 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:25,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:25,233 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:25,233 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:25,233 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:25,233 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:25,233 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:25,235 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,235 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,235 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:25,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:25,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:25,235 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,237 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,239 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:25,239 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:25,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:25,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:25,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:25,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57878 deadline: 1682042125239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:25,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57888 deadline: 1682042125239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,243 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,243 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,243 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,243 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,243 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:25,243 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,243 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:25,243 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:25,243 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,243 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:25,243 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:25,243 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:25,343 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:25,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:25,346 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:25,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:25,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:25,348 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:25,348 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:25,348 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:25,348 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:25,349 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,349 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,350 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:25,350 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:25,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:25,351 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,351 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:25,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:25,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:25,352 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:25,352 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:25,352 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:25,352 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:25,352 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:25,352 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:25,352 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:25,352 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:25,352 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:25,353 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:25,353 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:25,353 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,353 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:25,353 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:25,355 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,355 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:25,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:25,355 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,355 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:25,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:25,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:25,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:25,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:25,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b0b36b7[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:25,357 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:25,357 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,358 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,358 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:25,358 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:25,358 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:25,358 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,359 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:25,360 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:25,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:25,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:25,361 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:25,361 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:25,361 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:25,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:25,362 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:25,362 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:25,363 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:25,363 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:25,363 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:25,364 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,364 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,364 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:25,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:25,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:25,365 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:25,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,366 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,369 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,369 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,369 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:25,369 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:25,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:25,369 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,369 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:25,369 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,369 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,369 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,369 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:25,370 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:25,370 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:25,370 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:25,370 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:25,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:25,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:25,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:25,400 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421e1f97912952e44e4b761d2a041f14413 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421e1f97912952e44e4b761d2a041f14413 2023-04-21 01:54:25,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:25,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57878 deadline: 1682042125444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:25,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57888 deadline: 1682042125444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,471 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:25,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:25,473 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:25,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:25,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:25,474 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:25,474 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:25,474 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:25,474 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:25,475 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,475 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,476 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,476 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:25,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:25,477 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,477 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,478 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:25,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:25,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:25,478 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:25,478 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:25,478 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:25,478 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:25,478 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:25,478 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:25,479 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:25,479 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:25,479 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:25,479 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:25,479 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:25,479 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,479 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:25,479 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:25,481 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,481 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:25,481 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,481 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:25,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:25,483 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:25,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22365360[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:25,483 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,484 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,484 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:25,484 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:25,484 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:25,484 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,484 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:25,487 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:25,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:25,489 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:25,489 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:25,489 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:25,489 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:25,489 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:25,491 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,491 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:25,491 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:25,493 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:25,494 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,496 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,496 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,496 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,496 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,496 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:25,496 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,496 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:25,497 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:25,496 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,497 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,497 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:25,497 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:25,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:25,497 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:25,497 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:25,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:25,597 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:25,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:25,599 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:25,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:25,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:25,601 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:25,601 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:25,601 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:25,601 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:25,601 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,601 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,602 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,602 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:25,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:25,603 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,603 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:25,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:25,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:25,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:25,604 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:25,604 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:25,604 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:25,604 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:25,604 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:25,604 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:25,604 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:25,605 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:25,605 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:25,605 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:25,605 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:25,605 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,605 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:25,605 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:25,607 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:25,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:25,607 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:25,608 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,608 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:25,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:25,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:25,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:25,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f565e50[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:25,609 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:25,609 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,610 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,611 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:25,611 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:25,611 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,611 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:25,611 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:25,612 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,612 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:25,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:25,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:25,612 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:25,612 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:25,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:25,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:25,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:25,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:25,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:25,615 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:25,615 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:25,615 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:25,615 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:25,615 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:25,624 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,624 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:25,624 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:25,625 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:25,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:25,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:25,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,626 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:25,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,628 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,628 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,628 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,628 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:25,628 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,628 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:25,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:25,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:25,628 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:25,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:25,628 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:25,629 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:25,628 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,629 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,629 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,629 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:25,629 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:25,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:25,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:25,729 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:25,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:25,731 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:25,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:25,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:25,733 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:25,733 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:25,733 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:25,733 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:25,733 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,733 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,735 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:25,735 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:25,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:25,735 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,735 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:25,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:25,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:25,737 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:25,737 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:25,737 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:25,737 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:25,738 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:25,738 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:25,738 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:25,738 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:25,738 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:25,738 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:25,738 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:25,738 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:25,738 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:25,738 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,738 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:25,738 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:25,739 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,739 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:25,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:25,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:25,741 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,741 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:25,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:25,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:25,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:25,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fde2e52[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:25,742 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:25,742 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,744 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,744 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:25,744 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:25,744 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,745 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:25,745 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:25,746 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,746 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:25,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:25,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:25,746 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:25,746 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:25,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:25,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:25,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:25,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57888 deadline: 1682042125745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:25,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:25,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:25,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57878 deadline: 1682042125747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,749 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:25,749 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:25,749 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:25,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:25,749 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:25,749 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:25,750 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,750 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,750 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:25,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:25,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:25,751 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:25,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,752 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:25,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,754 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,754 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,754 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,754 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,754 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:25,754 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:25,754 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:25,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:25,754 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:25,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:25,755 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:25,754 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:25,754 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,755 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:25,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:25,755 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,755 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:25,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:25,807 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=193, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/078cdea4549549a3aa27cc652009de60 2023-04-21 01:54:25,855 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:25,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:25,857 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:25,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:25,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:25,858 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:25,858 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:25,858 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:25,858 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:25,859 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,859 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,865 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:25,865 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:25,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:25,865 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,865 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:25,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:25,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:25,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:25,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:25,873 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:25,873 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:25,873 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:25,874 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:25,874 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:25,874 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:25,874 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:25,876 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,876 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:25,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:25,876 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,876 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:25,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:25,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:25,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:25,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:25,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d320f90[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:25,879 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:25,879 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,880 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:25,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:25,880 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,881 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:25,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:25,882 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,882 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:25,882 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:25,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:25,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:25,882 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:25,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:25,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:25,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:25,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:25,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:25,885 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:25,885 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:25,885 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:25,885 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:25,885 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:25,889 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,889 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:25,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,889 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:25,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:25,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:25,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:25,890 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:25,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:25,891 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:25,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:25,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,894 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,894 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,894 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:25,894 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:25,894 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:25,894 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:25,894 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:25,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:25,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:25,894 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:25,894 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:25,894 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:25,894 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:25,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:25,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:25,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:25,894 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,895 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:25,895 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,994 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:25,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:25,997 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:25,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:25,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:25,998 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:25,998 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:25,998 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:25,998 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:25,999 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:25,999 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,000 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:26,000 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:26,000 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,000 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:26,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:26,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:26,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:26,002 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:26,002 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:26,002 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:26,002 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:26,002 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:26,002 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:26,002 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:26,002 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:26,002 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:26,002 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:26,002 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:26,002 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,002 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:26,002 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:26,004 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,004 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:26,004 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,004 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:26,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:26,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d0fea05[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:26,006 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:26,006 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,008 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,008 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:26,008 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:26,008 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:26,008 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,008 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:26,009 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,009 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:26,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:26,009 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:26,010 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:26,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:26,013 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:26,013 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:26,013 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:26,013 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:26,013 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:26,014 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:26,015 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,015 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:26,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:26,015 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,016 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,029 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:26,029 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:26,029 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:26,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:26,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:26,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:26,029 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:26,029 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:26,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:26,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:26,129 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:26,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:26,131 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:26,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:26,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:26,133 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:26,133 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:26,133 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:26,133 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:26,133 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,133 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,134 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,134 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:26,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:26,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:26,135 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,135 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:26,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:26,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:26,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:26,136 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:26,136 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:26,136 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:26,136 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:26,136 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:26,136 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:26,136 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:26,136 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:26,136 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:26,137 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:26,137 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:26,137 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,137 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:26,137 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:26,138 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,138 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:26,139 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,139 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:26,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:26,140 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:26,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66992a[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:26,140 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,141 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,141 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:26,141 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:26,141 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:26,141 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,142 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:26,144 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,144 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:26,144 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:26,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:26,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:26,144 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:26,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:26,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:26,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:26,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:26,146 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:26,147 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:26,147 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:26,147 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:26,147 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:26,148 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,148 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:26,148 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:26,149 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,150 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,153 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,153 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:26,153 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,153 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:26,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:26,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:26,153 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:26,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:26,153 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:26,153 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:26,153 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,153 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,153 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,153 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,153 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,154 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:26,154 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:26,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:26,221 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/661aa6fbaf9946a28e570db8d82263c3 2023-04-21 01:54:26,254 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:26,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:26,256 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:26,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:26,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57888 deadline: 1682042126250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:26,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:26,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:26,257 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:26,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57878 deadline: 1682042126254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,257 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:26,257 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:26,257 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:26,257 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,257 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,259 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,259 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:26,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:26,259 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,259 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:26,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:26,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:26,260 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:26,260 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:26,261 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:26,261 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:26,261 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:26,261 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:26,261 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:26,261 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:26,261 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:26,261 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:26,261 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:26,261 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,261 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:26,261 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:26,263 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,263 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:26,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:26,264 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,264 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:26,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:26,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:26,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:26,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:26,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eb49bca[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:26,265 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:26,265 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,267 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,267 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,267 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:26,267 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:26,267 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:26,267 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:26,268 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:26,268 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:26,268 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,268 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:26,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:26,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:26,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:26,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:26,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:26,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:26,271 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:26,271 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:26,271 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:26,271 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:26,271 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:26,273 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,273 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,273 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:26,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:26,274 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:26,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,275 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:26,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,278 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:26,278 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:26,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:26,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:26,278 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:26,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:26,278 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:26,278 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:26,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:26,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:26,279 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:26,279 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:26,280 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,280 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,280 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,280 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,280 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,280 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,280 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,379 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:26,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:26,381 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:26,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:26,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:26,384 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:26,384 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:26,384 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:26,384 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:26,385 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,385 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,386 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:26,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:26,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:26,386 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,387 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,387 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:26,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:26,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:26,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:26,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:26,388 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:26,388 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:26,388 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:26,392 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:26,392 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:26,392 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:26,392 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:26,392 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:26,392 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:26,392 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:26,392 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:26,392 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,392 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:26,392 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:26,394 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,394 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:26,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:26,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:26,394 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,394 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:26,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:26,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33b74503[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:26,398 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:26,398 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,399 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,399 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:26,399 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:26,400 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:26,400 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,400 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:26,403 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,403 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:26,403 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,403 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:26,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:26,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:26,405 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:26,405 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:26,406 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:26,406 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:26,406 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:26,407 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:26,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:26,407 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,407 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:26,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:26,408 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:26,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:26,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:26,409 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:26,414 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,414 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,414 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:26,414 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:26,414 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:26,414 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:26,414 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:26,414 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:26,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:26,414 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:26,414 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:26,414 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,415 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,415 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,415 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:26,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:26,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:26,515 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:26,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:26,517 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:26,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:26,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:26,520 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:26,521 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:26,521 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:26,521 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:26,521 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,521 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,523 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:26,523 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:26,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:26,524 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,524 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:26,525 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:26,525 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:26,525 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:26,525 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:26,525 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:26,525 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:26,525 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:26,525 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:26,525 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:26,525 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:26,525 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:26,525 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:26,525 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:26,525 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,525 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:26,525 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:26,527 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,527 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:26,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:26,527 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,527 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:26,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:26,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:26,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:26,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:26,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1daceb2[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:26,530 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:26,530 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,531 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,531 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:26,531 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:26,531 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:26,531 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,532 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:26,533 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:26,533 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:26,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:26,533 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:26,533 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:26,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:26,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:26,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:26,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:26,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:26,535 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:26,535 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:26,535 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:26,535 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:26,535 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:26,538 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,538 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:26,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:26,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,538 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:26,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:26,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:26,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:26,539 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:26,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,540 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:26,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:26,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,542 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,542 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:26,542 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:26,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:26,542 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,542 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,542 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:26,542 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:26,542 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:26,542 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:26,542 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:26,542 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:26,543 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,543 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:26,543 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:26,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:26,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:26,632 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/cf3fce13dfe348caaf9d67e444eb223e 2023-04-21 01:54:26,637 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/078cdea4549549a3aa27cc652009de60 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/078cdea4549549a3aa27cc652009de60 2023-04-21 01:54:26,641 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/078cdea4549549a3aa27cc652009de60, entries=200, sequenceid=193, filesize=32.1 K 2023-04-21 01:54:26,642 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/661aa6fbaf9946a28e570db8d82263c3 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/661aa6fbaf9946a28e570db8d82263c3 2023-04-21 01:54:26,643 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:26,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:26,645 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:26,649 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/661aa6fbaf9946a28e570db8d82263c3, entries=150, sequenceid=193, filesize=11.8 K 2023-04-21 01:54:26,650 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/cf3fce13dfe348caaf9d67e444eb223e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/cf3fce13dfe348caaf9d67e444eb223e 2023-04-21 01:54:26,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:26,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:26,656 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:26,656 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:26,656 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:26,656 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:26,656 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,657 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,660 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,660 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:26,660 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:26,666 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/cf3fce13dfe348caaf9d67e444eb223e, entries=150, sequenceid=193, filesize=11.8 K 2023-04-21 01:54:26,660 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:26,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:26,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,667 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~85.27 KB/87312, currentSize=154.31 KB/158010 for e00828c94a25954f7bdc6de3ac3e2662 in 1689ms, sequenceid=193, compaction requested=true 2023-04-21 01:54:26,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:26,667 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:26,667 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:26,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:26,668 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:26,667 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:26,668 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:26,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:26,668 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:26,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:26,669 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:26,669 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): e00828c94a25954f7bdc6de3ac3e2662/A is initiating minor compaction (all files) 2023-04-21 01:54:26,669 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33920 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:26,669 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e00828c94a25954f7bdc6de3ac3e2662/A in TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:26,669 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): e00828c94a25954f7bdc6de3ac3e2662/B is initiating minor compaction (all files) 2023-04-21 01:54:26,669 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/2b5f7650ffa942d083f5b4ecef724614, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/044e2cb029584348b6b441caf0adf245, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/078cdea4549549a3aa27cc652009de60] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp, totalSize=76.1 K 2023-04-21 01:54:26,669 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e00828c94a25954f7bdc6de3ac3e2662/B in TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:26,670 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/9d2b87195832418bb6c706bc49348be5, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/1a5fca224cb049a683942fe74af11f77, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/661aa6fbaf9946a28e570db8d82263c3] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp, totalSize=33.1 K 2023-04-21 01:54:26,670 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 2b5f7650ffa942d083f5b4ecef724614, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1682042062824 2023-04-21 01:54:26,670 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 9d2b87195832418bb6c706bc49348be5, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1682042062824 2023-04-21 01:54:26,670 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 044e2cb029584348b6b441caf0adf245, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1682042063452 2023-04-21 01:54:26,671 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 1a5fca224cb049a683942fe74af11f77, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1682042063452 2023-04-21 01:54:26,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:26,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:26,671 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 078cdea4549549a3aa27cc652009de60, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1682042064649 2023-04-21 01:54:26,671 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:26,671 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:26,671 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 661aa6fbaf9946a28e570db8d82263c3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1682042064669 2023-04-21 01:54:26,671 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:26,672 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:26,672 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:26,672 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:26,672 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:26,672 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e00828c94a25954f7bdc6de3ac3e2662 3/3 column families, dataSize=154.31 KB heapSize=428.16 KB 2023-04-21 01:54:26,672 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=A 2023-04-21 01:54:26,672 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:26,672 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=B 2023-04-21 01:54:26,673 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:26,673 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=C 2023-04-21 01:54:26,673 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:26,682 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): e00828c94a25954f7bdc6de3ac3e2662#B#compaction#170 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:26,699 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): e00828c94a25954f7bdc6de3ac3e2662#A#compaction#171 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:26,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:26,956 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:26,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:26,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042126967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:27,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042127072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,099 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304219f792ba51b78407aa72fd5dca4edc76a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304219f792ba51b78407aa72fd5dca4edc76a 2023-04-21 01:54:27,105 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/4490a21120c543e8b713afffedf40fc3 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/4490a21120c543e8b713afffedf40fc3 2023-04-21 01:54:27,105 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=281, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/f04311d5aada4476946982fd20be5f93 2023-04-21 01:54:27,111 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e00828c94a25954f7bdc6de3ac3e2662/B of e00828c94a25954f7bdc6de3ac3e2662 into 4490a21120c543e8b713afffedf40fc3(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:27,111 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:27,111 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., storeName=e00828c94a25954f7bdc6de3ac3e2662/B, priority=13, startTime=1682042066667; duration=0sec 2023-04-21 01:54:27,111 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:27,111 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:27,120 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33920 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:27,120 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): e00828c94a25954f7bdc6de3ac3e2662/C is initiating minor compaction (all files) 2023-04-21 01:54:27,120 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e00828c94a25954f7bdc6de3ac3e2662/C in TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:27,120 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/296a14adbbff4ebbb7e25742b414e2ec, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/98f4b7fcbb2245718e7454ab0747b10a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/cf3fce13dfe348caaf9d67e444eb223e] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp, totalSize=33.1 K 2023-04-21 01:54:27,121 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 296a14adbbff4ebbb7e25742b414e2ec, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1682042062824 2023-04-21 01:54:27,121 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 98f4b7fcbb2245718e7454ab0747b10a, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1682042063452 2023-04-21 01:54:27,122 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting cf3fce13dfe348caaf9d67e444eb223e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1682042064669 2023-04-21 01:54:27,130 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): e00828c94a25954f7bdc6de3ac3e2662#C#compaction#173 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:27,132 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/194a62ec98de40a09d9b467abb214060 2023-04-21 01:54:27,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:27,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57878 deadline: 1682042127265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:27,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57888 deadline: 1682042127267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:27,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042127275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,547 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/35b27328c5034d72925cdd99902c0912 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/35b27328c5034d72925cdd99902c0912 2023-04-21 01:54:27,554 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/7aeb45bed1cf4677803e7db243749cf7 2023-04-21 01:54:27,557 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e00828c94a25954f7bdc6de3ac3e2662/C of e00828c94a25954f7bdc6de3ac3e2662 into 35b27328c5034d72925cdd99902c0912(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:27,557 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:27,557 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., storeName=e00828c94a25954f7bdc6de3ac3e2662/C, priority=13, startTime=1682042066668; duration=0sec 2023-04-21 01:54:27,557 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:27,564 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/5d2e66e890c74a54986b07e6db554e31 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/5d2e66e890c74a54986b07e6db554e31 2023-04-21 01:54:27,564 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/f04311d5aada4476946982fd20be5f93 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/f04311d5aada4476946982fd20be5f93 2023-04-21 01:54:27,568 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/f04311d5aada4476946982fd20be5f93, entries=150, sequenceid=281, filesize=25.4 K 2023-04-21 01:54:27,569 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e00828c94a25954f7bdc6de3ac3e2662/A of e00828c94a25954f7bdc6de3ac3e2662 into 5d2e66e890c74a54986b07e6db554e31(size=25.6 K), total size for store is 51.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:27,569 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:27,569 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., storeName=e00828c94a25954f7bdc6de3ac3e2662/A, priority=13, startTime=1682042066667; duration=0sec 2023-04-21 01:54:27,569 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:27,569 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/194a62ec98de40a09d9b467abb214060 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/194a62ec98de40a09d9b467abb214060 2023-04-21 01:54:27,585 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/194a62ec98de40a09d9b467abb214060, entries=150, sequenceid=281, filesize=11.9 K 2023-04-21 01:54:27,586 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/7aeb45bed1cf4677803e7db243749cf7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/7aeb45bed1cf4677803e7db243749cf7 2023-04-21 01:54:27,598 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/7aeb45bed1cf4677803e7db243749cf7, entries=150, sequenceid=281, filesize=11.9 K 2023-04-21 01:54:27,599 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~428.11 KB/438384, currentSize=33.54 KB/34350 for e00828c94a25954f7bdc6de3ac3e2662 in 927ms, sequenceid=281, compaction requested=false 2023-04-21 01:54:27,599 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:27,599 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:27,599 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:27,599 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:27,599 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,599 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:27,599 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:27,601 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,601 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:27,601 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,601 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:27,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:27,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:27,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:27,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:27,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35c1b86a[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:27,603 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:27,603 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,605 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,606 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:27,606 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:27,606 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:27,606 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,606 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:27,607 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:27,607 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,607 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:27,607 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:27,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:27,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:27,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:27,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:27,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:27,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:27,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:27,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:27,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:27,611 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:27,611 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:27,611 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:27,611 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:27,611 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:27,613 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,613 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:27,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:27,613 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:27,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:27,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:27,614 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:27,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:27,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,615 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:27,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:27,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,618 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:27,618 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:27,618 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:27,619 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:27,619 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:27,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:27,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:27,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:27,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:27,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:27,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:27,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:27,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:27,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:27,621 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e00828c94a25954f7bdc6de3ac3e2662 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-21 01:54:27,621 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=A 2023-04-21 01:54:27,621 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:27,621 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=B 2023-04-21 01:54:27,621 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:27,621 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=C 2023-04-21 01:54:27,621 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:27,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682042067647, max timestamp=1682042067666Num uniques 859; 2023-04-21 01:54:27,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682042067647, max timestamp=1682042067666Num uniques 859; 2023-04-21 01:54:27,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682042067647, max timestamp=1682042067666Num uniques 859; 2023-04-21 01:54:27,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-21 01:54:27,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-21 01:54:27,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-21 01:54:27,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:27,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042127706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,719 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:27,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:27,721 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:27,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:27,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:27,722 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:27,722 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:27,722 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:27,722 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:27,723 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,723 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:27,725 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:27,725 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:27,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:27,725 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,725 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:27,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:27,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:27,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:27,726 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:27,726 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:27,726 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:27,726 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:27,726 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:27,726 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:27,726 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:27,726 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:27,726 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:27,727 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:27,727 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:27,727 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:27,727 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:27,727 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:27,727 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,727 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:27,727 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:27,728 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,728 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:27,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:27,729 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,729 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:27,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:27,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:27,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:27,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:27,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:27,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff11fd3[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:27,730 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:27,730 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,731 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,731 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:27,731 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,731 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:27,731 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:27,732 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:27,733 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:27,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:27,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:27,733 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:27,733 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:27,733 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:27,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:27,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:27,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:27,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:27,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:27,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:27,737 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:27,737 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:27,737 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:27,737 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:27,737 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:27,740 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,740 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:27,740 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:27,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:27,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:27,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:27,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:27,741 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:27,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:27,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:27,742 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:27,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:27,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,744 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,744 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:27,744 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:27,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:27,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:27,744 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:27,744 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:27,744 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:27,744 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,744 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,744 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,744 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,744 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:27,745 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:27,745 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:27,744 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:27,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:27,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:27,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:27,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:27,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042127808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,845 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:27,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:27,847 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:27,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:27,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:27,848 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:27,848 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:27,848 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:27,848 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:27,849 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,849 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:27,850 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,850 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:27,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:27,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:27,850 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,850 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:27,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:27,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:27,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:27,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:27,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:27,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:27,851 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:27,851 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:27,851 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:27,852 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:27,852 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:27,852 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:27,852 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:27,852 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:27,852 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:27,852 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:27,852 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:27,852 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,852 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:27,852 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:27,856 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,856 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:27,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:27,856 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,856 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:27,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:27,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:27,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:27,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:27,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:27,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9a5c1e2[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:27,858 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:27,858 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,859 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,859 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:27,859 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:27,859 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:27,859 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,859 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:27,860 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:27,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:27,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:27,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:27,861 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:27,861 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:27,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:27,861 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:27,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:27,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:27,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:27,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:27,863 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:27,863 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:27,863 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:27,863 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:27,863 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:27,864 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,864 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:27,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,865 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:27,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:27,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:27,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:27,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:27,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,866 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:27,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:27,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,867 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,869 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,869 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:27,886 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:27,886 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,886 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,886 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,886 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:27,886 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:27,886 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:27,886 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:27,886 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:27,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:27,886 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:27,888 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:27,888 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:27,988 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:27,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:27,990 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:27,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:27,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:27,991 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:27,991 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:27,991 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:27,991 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:27,992 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,992 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:27,993 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,993 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:27,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:27,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:27,993 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,994 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:27,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:27,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:27,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:27,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:27,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:27,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:27,995 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:27,995 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:27,995 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:27,995 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:27,995 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:27,995 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:27,995 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:27,995 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:27,995 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:27,995 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:27,996 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:27,996 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,996 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:27,996 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:27,998 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,999 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:27,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:27,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:27,999 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:27,999 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:27,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:27,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:28,001 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:28,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e4d4882[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:28,001 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,002 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,002 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:28,002 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,002 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:28,002 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:28,002 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:28,004 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,004 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:28,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:28,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:28,004 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:28,004 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:28,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:28,006 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:28,006 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:28,006 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:28,006 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:28,006 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:28,008 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,008 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,008 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:28,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:28,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:28,009 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,010 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,012 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,012 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,012 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,012 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:28,012 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:28,013 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,013 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:28,013 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:28,012 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:28,012 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:28,013 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,013 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:28,013 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:28,013 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:28,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:28,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:28,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:28,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042128011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,048 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304213f4bc8537518420f8eac24c2667ec2d4 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304213f4bc8537518420f8eac24c2667ec2d4 2023-04-21 01:54:28,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:28,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57852 deadline: 1682042128100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,103 DEBUG [Thread-1020] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8173 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:54:28,113 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:28,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:28,115 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:28,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:28,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:28,117 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:28,117 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:28,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:28,117 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:28,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57864 deadline: 1682042128115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,117 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:28,118 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,118 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,118 DEBUG [Thread-1014] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8189 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:54:28,120 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:28,120 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:28,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:28,120 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,120 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:28,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:28,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:28,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:28,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:28,121 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:28,121 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:28,121 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:28,121 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:28,121 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:28,121 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:28,121 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:28,121 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:28,121 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:28,121 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:28,121 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:28,121 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,121 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:28,121 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:28,124 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,124 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:28,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:28,124 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,124 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:28,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:28,126 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:28,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36f1f970[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:28,126 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,127 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,127 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:28,127 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:28,127 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,127 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:28,127 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:28,128 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,128 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:28,128 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:28,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:28,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:28,128 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:28,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:28,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:28,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:28,132 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:28,132 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:28,132 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:28,132 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:28,132 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:28,133 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,133 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:28,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:28,133 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:28,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:28,134 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:28,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:28,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,137 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,141 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,141 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:28,141 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:28,141 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:28,141 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:28,141 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:28,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:28,141 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:28,141 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,141 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,141 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,141 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,141 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,141 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:28,142 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:28,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:28,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:28,242 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:28,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:28,244 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:28,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:28,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:28,245 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:28,245 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:28,245 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:28,245 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:28,246 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,246 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,247 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,247 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:28,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:28,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:28,247 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,247 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:28,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:28,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:28,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:28,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:28,250 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:28,250 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:28,250 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:28,250 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:28,250 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:28,250 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:28,250 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:28,251 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:28,251 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:28,251 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:28,251 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:28,251 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,251 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:28,251 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:28,252 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,252 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:28,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:28,253 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,253 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:28,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:28,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:28,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eccaf48[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:28,254 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:28,254 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,259 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,259 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,259 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:28,259 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:28,259 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:28,259 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:28,260 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:28,261 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:28,260 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,261 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:28,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:28,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:28,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:28,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:28,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:28,263 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:28,263 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:28,263 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:28,263 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:28,263 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:28,264 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,264 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,264 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:28,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:28,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:28,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:28,265 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,266 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,268 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,268 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,268 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:28,268 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:28,268 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:28,268 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:28,268 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:28,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:28,268 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,268 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:28,268 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,268 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:28,268 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,268 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:28,268 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:28,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:28,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:28,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:28,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042128318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,369 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:28,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:28,371 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:28,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:28,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:28,372 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:28,372 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:28,372 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:28,372 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:28,372 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,373 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,374 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:28,374 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:28,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:28,375 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,375 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:28,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:28,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:28,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:28,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:28,376 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:28,376 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:28,376 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:28,376 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:28,376 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:28,376 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:28,376 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:28,376 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:28,376 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:28,377 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:28,377 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:28,377 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,377 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:28,377 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:28,378 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,378 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:28,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:28,378 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,378 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:28,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:28,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:28,380 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:28,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5941dba2[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:28,380 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,381 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,381 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:28,381 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:28,381 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:28,381 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,381 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:28,382 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:28,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:28,382 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:28,382 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:28,382 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:28,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:28,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:28,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:28,385 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:28,385 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:28,385 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:28,385 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:28,385 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:28,388 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,388 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:28,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,388 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:28,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:28,388 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,390 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,392 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,392 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,392 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,392 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:28,392 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:28,392 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:28,392 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:28,392 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,392 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:28,392 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,392 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:28,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:28,392 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,392 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:28,392 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:28,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:28,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:28,454 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=294, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/c5c9a827457042db9ded9cb7e634c152 2023-04-21 01:54:28,493 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:28,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:28,495 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:28,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:28,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:28,496 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:28,496 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:28,496 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:28,496 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:28,497 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,497 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,498 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:28,498 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:28,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:28,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,498 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,498 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:28,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:28,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:28,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:28,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:28,500 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:28,500 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:28,500 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:28,501 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:28,501 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:28,501 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:28,501 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:28,501 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:28,501 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:28,501 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:28,501 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:28,501 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,501 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:28,501 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:28,503 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,503 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:28,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:28,503 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:28,503 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:28,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:28,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dcc8150[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:28,504 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:28,505 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,506 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,506 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:28,506 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:28,506 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,506 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:28,506 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:28,509 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:28,509 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:28,509 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,509 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:28,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:28,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:28,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:28,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:28,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:28,511 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:28,511 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:28,511 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:28,511 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:28,511 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:28,513 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,513 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:28,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:28,513 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:28,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:28,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:28,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:28,513 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,515 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,517 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,517 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:28,517 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:28,517 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:28,517 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:28,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:28,517 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:28,517 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:28,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:28,517 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,517 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,517 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,517 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,517 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,517 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:28,517 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:28,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:28,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:28,617 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:28,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:28,619 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:28,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:28,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:28,621 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:28,621 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:28,621 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:28,621 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:28,621 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,621 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,622 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:28,622 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:28,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:28,623 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,623 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:28,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:28,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:28,624 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:28,624 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:28,624 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:28,624 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:28,624 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:28,624 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:28,624 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:28,624 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:28,624 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:28,624 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:28,624 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:28,624 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,624 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:28,624 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:28,627 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,627 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:28,627 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,628 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:28,629 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:28,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64cfd9e0[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:28,629 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,630 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,630 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:28,630 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:28,630 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:28,631 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,631 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:28,631 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,631 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:28,631 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:28,631 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:28,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:28,634 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:28,634 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:28,634 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:28,634 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:28,634 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:28,639 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:28,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:28,639 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,639 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:28,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:28,640 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:28,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:28,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,641 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,643 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:28,643 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:28,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:28,643 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,643 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:28,643 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:28,643 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,643 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,643 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,643 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,643 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,643 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:28,643 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:28,643 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:28,643 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:28,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:28,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:28,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:28,743 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:28,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:28,746 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:28,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:28,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:28,747 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:28,747 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:28,747 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:28,747 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:28,747 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,747 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,749 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:28,749 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:28,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:28,749 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,749 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:28,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:28,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:28,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:28,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:28,750 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:28,750 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:28,750 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:28,751 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:28,751 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:28,751 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:28,751 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:28,751 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:28,751 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:28,751 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:28,751 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:28,751 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,751 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:28,751 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:28,752 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,752 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:28,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:28,753 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,753 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:28,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:28,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:28,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f629cde[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:28,754 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:28,754 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,758 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,758 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:28,758 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:28,758 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:28,758 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,758 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:28,760 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,760 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:28,760 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:28,760 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:28,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:28,763 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:28,763 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:28,763 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:28,763 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:28,763 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:28,764 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,764 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:28,764 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:28,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:28,765 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,766 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,768 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,769 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,769 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,769 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,769 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:28,769 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,769 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:28,769 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:28,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:28,769 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:28,769 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,769 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:28,769 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:28,769 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:28,769 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:28,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:28,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57904 deadline: 1682042128826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,864 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/a284ace671164f8dbfde7eafb3bf950a 2023-04-21 01:54:28,869 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:28,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:28,872 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:28,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:28,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:28,874 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:28,874 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:28,874 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:28,874 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:28,875 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,875 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,877 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:28,877 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:28,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:28,877 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,877 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:28,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:28,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,878 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:28,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:28,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:28,879 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:28,879 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:28,879 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:28,879 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:28,879 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:28,879 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:28,879 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:28,879 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:28,879 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:28,879 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:28,879 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:28,879 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,879 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:28,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:28,882 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,882 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:28,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:28,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:28,882 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,882 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:28,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:28,884 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:28,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28a9d45b[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:28,884 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,885 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,885 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:28,885 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:28,886 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:28,886 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,886 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:28,887 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,887 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:28,887 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:28,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:28,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:28,887 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:28,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:28,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:28,889 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:28,890 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:28,890 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:28,890 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:28,890 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:28,893 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,893 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:28,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:28,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,893 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:28,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:28,894 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,895 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,897 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,897 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,897 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:28,897 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:28,897 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,897 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:28,897 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:28,897 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:28,897 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:28,897 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:28,897 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:28,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:28,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:28,897 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:28,897 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:28,898 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:28,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:28,998 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:28,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:29,000 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:29,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:29,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:29,001 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:29,001 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:29,001 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:29,001 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:29,002 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,002 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,003 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:29,003 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:29,003 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,003 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:29,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:29,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:29,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:29,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:29,005 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:29,005 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:29,005 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:29,005 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:29,005 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:29,005 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:29,005 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:29,005 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:29,005 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:29,005 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:29,005 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:29,005 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,005 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:29,005 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:29,007 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,007 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:29,007 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,007 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:29,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:29,008 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:29,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11d6061f[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:29,008 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,011 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,011 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:29,011 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,011 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:29,011 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:29,011 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:29,012 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,012 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:29,012 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:29,012 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:29,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:29,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:29,015 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:29,015 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:29,015 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:29,015 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:29,015 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:29,017 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,017 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,017 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:29,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:29,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:29,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:29,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:29,017 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:29,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:29,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,018 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,027 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,027 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,027 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,027 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:29,027 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:29,027 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:29,027 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:29,027 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,027 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,027 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,027 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:29,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:29,027 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:29,027 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:29,028 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:29,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:29,128 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:29,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:29,130 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:29,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:29,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:29,131 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:29,131 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:29,131 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:29,131 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:29,132 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,132 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,133 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:29,133 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,133 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,133 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:29,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:29,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:29,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:29,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:29,134 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:29,134 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:29,134 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:29,134 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:29,134 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:29,134 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:29,135 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:29,135 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:29,135 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:29,135 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:29,135 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:29,135 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,135 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:29,135 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:29,136 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,136 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:29,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:29,137 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,137 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:29,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:29,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:29,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:29,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:29,138 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:29,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b4cdd1c[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:29,138 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,139 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,140 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:29,140 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:29,140 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,140 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:29,140 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:29,142 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,142 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:29,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:29,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:29,142 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:29,143 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:29,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:29,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:29,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:29,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:29,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:29,145 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:29,145 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:29,145 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:29,145 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:29,145 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:29,147 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,147 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:29,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,147 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:29,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:29,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:29,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:29,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:29,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,149 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:29,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,154 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,157 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,157 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,157 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,157 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,157 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:29,157 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:29,157 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,157 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:29,157 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,157 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:29,157 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:29,157 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,157 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:29,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:29,157 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:29,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:29,258 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:29,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:29,260 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:29,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:29,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:29,261 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:29,261 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:29,261 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:29,261 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:29,262 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,262 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,263 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:29,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:29,263 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,263 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,264 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:29,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,264 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:29,264 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:29,264 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:29,264 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:29,264 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:29,265 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:29,265 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:29,265 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:29,265 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:29,265 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:29,265 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:29,265 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:29,265 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:29,265 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:29,265 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,265 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:29,265 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:29,268 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,268 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:29,268 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,268 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:29,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:29,270 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:29,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@602324c7[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:29,270 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,271 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:29,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:29,271 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,271 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:29,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:29,273 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:29,273 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:29,273 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,273 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:29,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:29,275 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:29,275 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:29,275 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:29,275 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:29,275 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:29,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:29,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57888 deadline: 1682042129274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,277 DEBUG [Thread-1022] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:54:29,277 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,277 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:29,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:29,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,278 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:29,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:29,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:29,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:29,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,279 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:29,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,280 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:29,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,281 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/454a6881f23e42e4b6148506c8e1c45c 2023-04-21 01:54:29,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,284 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:29,284 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,284 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:29,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:29,284 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:29,284 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:29,284 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:29,284 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,284 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,284 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,284 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,284 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,284 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,284 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:29,284 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:29,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:29,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:29,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:29,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:29,286 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/c5c9a827457042db9ded9cb7e634c152 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/c5c9a827457042db9ded9cb7e634c152 2023-04-21 01:54:29,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:29,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57878 deadline: 1682042129283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,288 DEBUG [Thread-1016] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:54:29,291 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/c5c9a827457042db9ded9cb7e634c152, entries=200, sequenceid=294, filesize=32.3 K 2023-04-21 01:54:29,292 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/a284ace671164f8dbfde7eafb3bf950a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/a284ace671164f8dbfde7eafb3bf950a 2023-04-21 01:54:29,295 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/a284ace671164f8dbfde7eafb3bf950a, entries=150, sequenceid=294, filesize=11.9 K 2023-04-21 01:54:29,296 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/454a6881f23e42e4b6148506c8e1c45c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/454a6881f23e42e4b6148506c8e1c45c 2023-04-21 01:54:29,300 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/454a6881f23e42e4b6148506c8e1c45c, entries=150, sequenceid=294, filesize=11.9 K 2023-04-21 01:54:29,300 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e00828c94a25954f7bdc6de3ac3e2662 in 1679ms, sequenceid=294, compaction requested=true 2023-04-21 01:54:29,300 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:29,300 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:29,301 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:29,301 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:29,301 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:29,301 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:29,302 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85336 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:29,302 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): e00828c94a25954f7bdc6de3ac3e2662/A is initiating minor compaction (all files) 2023-04-21 01:54:29,302 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:29,302 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e00828c94a25954f7bdc6de3ac3e2662/A in TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:29,302 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): e00828c94a25954f7bdc6de3ac3e2662/B is initiating minor compaction (all files) 2023-04-21 01:54:29,302 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/5d2e66e890c74a54986b07e6db554e31, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/f04311d5aada4476946982fd20be5f93, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/c5c9a827457042db9ded9cb7e634c152] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp, totalSize=83.3 K 2023-04-21 01:54:29,302 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e00828c94a25954f7bdc6de3ac3e2662/B in TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:29,302 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/4490a21120c543e8b713afffedf40fc3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/194a62ec98de40a09d9b467abb214060, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/a284ace671164f8dbfde7eafb3bf950a] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp, totalSize=35.9 K 2023-04-21 01:54:29,302 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 5d2e66e890c74a54986b07e6db554e31, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1682042064669 2023-04-21 01:54:29,302 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 4490a21120c543e8b713afffedf40fc3, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1682042064669 2023-04-21 01:54:29,303 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting f04311d5aada4476946982fd20be5f93, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1682042065124 2023-04-21 01:54:29,304 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 194a62ec98de40a09d9b467abb214060, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1682042065124 2023-04-21 01:54:29,304 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting c5c9a827457042db9ded9cb7e634c152, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1682042066963 2023-04-21 01:54:29,304 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting a284ace671164f8dbfde7eafb3bf950a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1682042066963 2023-04-21 01:54:29,320 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): e00828c94a25954f7bdc6de3ac3e2662#B#compaction#179 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:29,322 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): e00828c94a25954f7bdc6de3ac3e2662#A#compaction#178 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:29,384 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:29,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:29,387 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:29,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:29,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:29,388 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:29,388 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:29,388 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:29,388 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:29,388 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,389 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,390 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,390 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:29,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:29,390 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,390 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:29,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:29,391 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:29,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:29,391 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:29,391 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:29,391 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:29,392 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:29,392 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:29,392 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:29,392 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:29,392 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e00828c94a25954f7bdc6de3ac3e2662 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-21 01:54:29,392 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=A 2023-04-21 01:54:29,392 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:29,392 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=B 2023-04-21 01:54:29,393 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:29,393 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=C 2023-04-21 01:54:29,393 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:29,416 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421c44ccee88ec94ed2b11b48b841ebfaa6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421c44ccee88ec94ed2b11b48b841ebfaa6 2023-04-21 01:54:29,426 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=329, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/41a3de37d1aa47759fba629083b008a8 2023-04-21 01:54:29,435 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/fc00e37fa6234c94bf0d82cf5e2bc213 2023-04-21 01:54:29,446 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/e1d2ab863c9547b7a4ed66c3c859bb20 2023-04-21 01:54:29,456 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/41a3de37d1aa47759fba629083b008a8 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/41a3de37d1aa47759fba629083b008a8 2023-04-21 01:54:29,460 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/41a3de37d1aa47759fba629083b008a8, entries=150, sequenceid=329, filesize=25.4 K 2023-04-21 01:54:29,461 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/fc00e37fa6234c94bf0d82cf5e2bc213 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/fc00e37fa6234c94bf0d82cf5e2bc213 2023-04-21 01:54:29,465 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/fc00e37fa6234c94bf0d82cf5e2bc213, entries=150, sequenceid=329, filesize=11.9 K 2023-04-21 01:54:29,466 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/e1d2ab863c9547b7a4ed66c3c859bb20 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/e1d2ab863c9547b7a4ed66c3c859bb20 2023-04-21 01:54:29,470 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/e1d2ab863c9547b7a4ed66c3c859bb20, entries=150, sequenceid=329, filesize=11.9 K 2023-04-21 01:54:29,471 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=0 B/0 for e00828c94a25954f7bdc6de3ac3e2662 in 79ms, sequenceid=329, compaction requested=true 2023-04-21 01:54:29,471 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:29,471 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:29,471 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:29,471 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:29,471 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,471 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:29,471 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:29,473 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,473 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:29,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:29,473 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,473 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:29,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:29,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:29,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:29,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:29,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ce2f94[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:29,475 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:29,475 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,476 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,476 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:29,476 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:29,476 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:29,476 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,476 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:29,478 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,478 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:29,478 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:29,479 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:29,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:29,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:29,482 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:29,482 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:29,482 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:29,482 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:29,482 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:29,484 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,484 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,484 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:29,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:29,485 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:29,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,486 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,491 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,491 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,491 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,491 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,492 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,492 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,492 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,492 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:29,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:29,492 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:29,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:29,492 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:29,492 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:29,492 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:29,492 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:29,492 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:29,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:29,592 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:29,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:29,595 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:29,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:29,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:29,597 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:29,597 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:29,597 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:29,597 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:29,597 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,597 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,599 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:29,599 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:29,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:29,600 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,600 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:29,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:29,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:29,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:29,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:29,601 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:29,601 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:29,601 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:29,602 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:29,602 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:29,602 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:29,602 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:29,602 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:29,602 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:29,602 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:29,602 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:29,602 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,602 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:29,602 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:29,603 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,603 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:29,604 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,604 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:29,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:29,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:29,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@278927ac[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:29,606 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:29,606 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,607 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,607 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:29,607 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:29,607 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:29,607 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,607 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:29,610 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,610 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:29,611 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:29,611 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:29,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:29,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:29,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:29,626 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:29,626 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:29,626 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:29,626 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:29,626 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:29,628 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,628 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,628 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:29,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:29,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:29,629 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:29,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,631 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,633 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,633 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:29,633 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:29,633 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:29,633 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,633 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:29,633 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:29,633 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:29,633 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,634 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,634 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,634 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:29,634 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:29,634 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:29,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:29,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:29,737 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:29,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:29,739 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/0ccd49879c3b4f44a069de4fbd971a96 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/0ccd49879c3b4f44a069de4fbd971a96 2023-04-21 01:54:29,741 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:29,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:29,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:29,742 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:29,742 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:29,742 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:29,742 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:29,743 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,743 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,744 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,744 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:29,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:29,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:29,744 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,744 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,746 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e00828c94a25954f7bdc6de3ac3e2662/B of e00828c94a25954f7bdc6de3ac3e2662 into 0ccd49879c3b4f44a069de4fbd971a96(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:29,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:29,746 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:29,746 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., storeName=e00828c94a25954f7bdc6de3ac3e2662/B, priority=13, startTime=1682042069301; duration=0sec 2023-04-21 01:54:29,746 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:29,746 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:54:29,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:29,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:29,747 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:29,747 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:29,747 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:29,747 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:29,747 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:29,747 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:29,747 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:29,747 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:29,747 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:29,748 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:29,748 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:29,748 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48902 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:54:29,748 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,748 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): e00828c94a25954f7bdc6de3ac3e2662/C is initiating minor compaction (all files) 2023-04-21 01:54:29,748 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:29,748 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:29,748 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e00828c94a25954f7bdc6de3ac3e2662/C in TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:29,748 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/35b27328c5034d72925cdd99902c0912, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/7aeb45bed1cf4677803e7db243749cf7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/454a6881f23e42e4b6148506c8e1c45c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/e1d2ab863c9547b7a4ed66c3c859bb20] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp, totalSize=47.8 K 2023-04-21 01:54:29,749 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 35b27328c5034d72925cdd99902c0912, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1682042064669 2023-04-21 01:54:29,749 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 7aeb45bed1cf4677803e7db243749cf7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1682042065124 2023-04-21 01:54:29,750 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 454a6881f23e42e4b6148506c8e1c45c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1682042066963 2023-04-21 01:54:29,750 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,750 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:29,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:29,750 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting e1d2ab863c9547b7a4ed66c3c859bb20, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682042067685 2023-04-21 01:54:29,750 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,750 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:29,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:29,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:29,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ac0ba7c[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:29,753 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:29,753 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,755 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,755 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:29,755 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:29,755 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:29,755 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,755 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:29,757 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,757 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:29,757 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:29,757 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:29,759 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): e00828c94a25954f7bdc6de3ac3e2662#C#compaction#183 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:29,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:29,761 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:29,761 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:29,761 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:29,761 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:29,761 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:29,763 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,763 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,763 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:29,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,763 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:29,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,765 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,769 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,769 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,769 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,769 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,769 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:29,769 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:29,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:29,769 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:29,769 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:29,769 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:29,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:29,769 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:29,769 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:29,769 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,770 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:29,770 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:29,870 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:29,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:29,872 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:29,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:29,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:29,874 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:29,874 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:29,874 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:29,874 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:29,874 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,874 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,876 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:29,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:29,876 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,877 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:29,877 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:29,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:29,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:29,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:29,878 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:29,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:29,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:29,879 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:29,879 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:29,879 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:29,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:29,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:29,880 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:29,880 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:29,880 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e00828c94a25954f7bdc6de3ac3e2662 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-21 01:54:29,880 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=A 2023-04-21 01:54:29,880 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:29,881 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=B 2023-04-21 01:54:29,881 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:29,881 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=C 2023-04-21 01:54:29,881 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:29,906 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042122d3ecc3ba4844889fa791faa261bdfc to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042122d3ecc3ba4844889fa791faa261bdfc 2023-04-21 01:54:29,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:29,911 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:29,917 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=337, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/84b9ef0e620640e68de465b83f6a907f 2023-04-21 01:54:29,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042069939, max timestamp=1682042069953Num uniques 859; 2023-04-21 01:54:29,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:54:29,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042069939, max timestamp=1682042069953Num uniques 859; 2023-04-21 01:54:29,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:54:29,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042069939, max timestamp=1682042069953Num uniques 859; 2023-04-21 01:54:29,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:54:30,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682042069985, max timestamp=1682042069998Num uniques 1018; 2023-04-21 01:54:30,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682042069985, max timestamp=1682042069998Num uniques 1018; 2023-04-21 01:54:30,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682042069985, max timestamp=1682042069998Num uniques 1018; 2023-04-21 01:54:30,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-21 01:54:30,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-21 01:54:30,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-21 01:54:30,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682042070026, max timestamp=1682042070046Num uniques 1077; 2023-04-21 01:54:30,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-21 01:54:30,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682042070026, max timestamp=1682042070046Num uniques 1077; 2023-04-21 01:54:30,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-21 01:54:30,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682042070026, max timestamp=1682042070046Num uniques 1077; 2023-04-21 01:54:30,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-21 01:54:30,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682042070089, max timestamp=1682042070105Num uniques 1136; 2023-04-21 01:54:30,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-21 01:54:30,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682042070089, max timestamp=1682042070105Num uniques 1136; 2023-04-21 01:54:30,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-21 01:54:30,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682042070089, max timestamp=1682042070105Num uniques 1136; 2023-04-21 01:54:30,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-21 01:54:30,152 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/564a5beb388b4e49a2cda44823c00ae6 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/564a5beb388b4e49a2cda44823c00ae6 2023-04-21 01:54:30,156 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e00828c94a25954f7bdc6de3ac3e2662/A of e00828c94a25954f7bdc6de3ac3e2662 into 564a5beb388b4e49a2cda44823c00ae6(size=25.8 K), total size for store is 51.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:30,156 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:30,156 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., storeName=e00828c94a25954f7bdc6de3ac3e2662/A, priority=13, startTime=1682042069300; duration=0sec 2023-04-21 01:54:30,156 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:30,171 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/59c5fe3771794beb829e7d66b93175f7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/59c5fe3771794beb829e7d66b93175f7 2023-04-21 01:54:30,176 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e00828c94a25954f7bdc6de3ac3e2662/C of e00828c94a25954f7bdc6de3ac3e2662 into 59c5fe3771794beb829e7d66b93175f7(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:30,176 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:30,176 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., storeName=e00828c94a25954f7bdc6de3ac3e2662/C, priority=12, startTime=1682042069301; duration=0sec 2023-04-21 01:54:30,177 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:30,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1682042070132, max timestamp=1682042070176Num uniques 1095; 2023-04-21 01:54:30,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-21 01:54:30,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1682042070132, max timestamp=1682042070176Num uniques 1095; 2023-04-21 01:54:30,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-21 01:54:30,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1682042070132, max timestamp=1682042070176Num uniques 1095; 2023-04-21 01:54:30,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-21 01:54:30,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682042070213, max timestamp=1682042070223Num uniques 1154; 2023-04-21 01:54:30,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=9340, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-04-21 01:54:30,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682042070213, max timestamp=1682042070223Num uniques 1154; 2023-04-21 01:54:30,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=9340, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-04-21 01:54:30,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682042070213, max timestamp=1682042070223Num uniques 1154; 2023-04-21 01:54:30,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=9340, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-04-21 01:54:30,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682042070250, max timestamp=1682042070273Num uniques 1063; 2023-04-21 01:54:30,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-21 01:54:30,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682042070250, max timestamp=1682042070273Num uniques 1063; 2023-04-21 01:54:30,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-21 01:54:30,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682042070250, max timestamp=1682042070273Num uniques 1063; 2023-04-21 01:54:30,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-21 01:54:30,334 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/feae6c9c5f8248938aa23dd5ae7804ec 2023-04-21 01:54:30,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682042070324, max timestamp=1682042070333Num uniques 1072; 2023-04-21 01:54:30,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682042070324, max timestamp=1682042070333Num uniques 1072; 2023-04-21 01:54:30,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682042070324, max timestamp=1682042070333Num uniques 1072; 2023-04-21 01:54:30,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-21 01:54:30,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-21 01:54:30,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-21 01:54:30,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682042070374, max timestamp=1682042070392Num uniques 1081; 2023-04-21 01:54:30,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682042070374, max timestamp=1682042070392Num uniques 1081; 2023-04-21 01:54:30,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-21 01:54:30,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682042070374, max timestamp=1682042070392Num uniques 1081; 2023-04-21 01:54:30,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-21 01:54:30,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-21 01:54:30,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1682042070434, max timestamp=1682042070451Num uniques 1040; 2023-04-21 01:54:30,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-21 01:54:30,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1682042070434, max timestamp=1682042070451Num uniques 1090; 2023-04-21 01:54:30,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=13280, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-04-21 01:54:30,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1682042070434, max timestamp=1682042070451Num uniques 1090; 2023-04-21 01:54:30,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=13280, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-04-21 01:54:30,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1682042070467, max timestamp=1682042070503Num uniques 1149; 2023-04-21 01:54:30,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=13694, suffix heap size=147464,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=299 2023-04-21 01:54:30,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1682042070467, max timestamp=1682042070503Num uniques 1149; 2023-04-21 01:54:30,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=13694, suffix heap size=147464,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=299 2023-04-21 01:54:30,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1682042070467, max timestamp=1682042070503Num uniques 1149; 2023-04-21 01:54:30,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=13694, suffix heap size=147464,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=299 2023-04-21 01:54:30,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682042070536, max timestamp=1682042070564Num uniques 1158; 2023-04-21 01:54:30,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=9520, suffix heap size=148400,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=208 2023-04-21 01:54:30,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682042070536, max timestamp=1682042070564Num uniques 1158; 2023-04-21 01:54:30,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=9520, suffix heap size=148400,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=208 2023-04-21 01:54:30,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682042070536, max timestamp=1682042070564Num uniques 1158; 2023-04-21 01:54:30,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=9520, suffix heap size=148400,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=208 2023-04-21 01:54:30,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1682042070589, max timestamp=1682042070623Num uniques 1067; 2023-04-21 01:54:30,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-21 01:54:30,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1682042070589, max timestamp=1682042070623Num uniques 1067; 2023-04-21 01:54:30,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1682042070589, max timestamp=1682042070623Num uniques 1067; 2023-04-21 01:54:30,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-21 01:54:30,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-21 01:54:30,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682042070644, max timestamp=1682042070666Num uniques 1026; 2023-04-21 01:54:30,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-21 01:54:30,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682042070644, max timestamp=1682042070666Num uniques 1026; 2023-04-21 01:54:30,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-21 01:54:30,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682042070644, max timestamp=1682042070666Num uniques 1026; 2023-04-21 01:54:30,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-21 01:54:30,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682042070690, max timestamp=1682042070707Num uniques 1035; 2023-04-21 01:54:30,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-21 01:54:30,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682042070690, max timestamp=1682042070707Num uniques 1035; 2023-04-21 01:54:30,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-21 01:54:30,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682042070690, max timestamp=1682042070707Num uniques 1035; 2023-04-21 01:54:30,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-21 01:54:30,751 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/7f8ada7d5bd748eaa5ca3675217885ec 2023-04-21 01:54:30,757 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/84b9ef0e620640e68de465b83f6a907f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/84b9ef0e620640e68de465b83f6a907f 2023-04-21 01:54:30,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1682042070736, max timestamp=1682042070754Num uniques 1094; 2023-04-21 01:54:30,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=13464, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-04-21 01:54:30,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1682042070736, max timestamp=1682042070754Num uniques 1094; 2023-04-21 01:54:30,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=13464, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-04-21 01:54:30,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1682042070736, max timestamp=1682042070754Num uniques 1094; 2023-04-21 01:54:30,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=13464, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-04-21 01:54:30,761 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/84b9ef0e620640e68de465b83f6a907f, entries=100, sequenceid=337, filesize=18.6 K 2023-04-21 01:54:30,762 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/feae6c9c5f8248938aa23dd5ae7804ec as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/feae6c9c5f8248938aa23dd5ae7804ec 2023-04-21 01:54:30,767 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/feae6c9c5f8248938aa23dd5ae7804ec, entries=100, sequenceid=337, filesize=9.5 K 2023-04-21 01:54:30,768 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/7f8ada7d5bd748eaa5ca3675217885ec as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/7f8ada7d5bd748eaa5ca3675217885ec 2023-04-21 01:54:30,772 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/7f8ada7d5bd748eaa5ca3675217885ec, entries=100, sequenceid=337, filesize=9.5 K 2023-04-21 01:54:30,773 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=67.09 KB/68700 for e00828c94a25954f7bdc6de3ac3e2662 in 893ms, sequenceid=337, compaction requested=true 2023-04-21 01:54:30,773 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:30,773 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:30,773 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:30,773 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:30,773 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:30,774 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e00828c94a25954f7bdc6de3ac3e2662 3/3 column families, dataSize=73.80 KB heapSize=193.03 KB 2023-04-21 01:54:30,774 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:30,774 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:30,774 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=A 2023-04-21 01:54:30,774 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:30,774 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=B 2023-04-21 01:54:30,774 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:30,774 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=C 2023-04-21 01:54:30,774 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:30,775 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:30,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:30,776 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:30,776 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:30,776 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:30,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:30,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:30,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:30,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:30,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:30,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33bcbaff[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:30,778 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:30,778 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:30,779 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:30,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:30,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:30,780 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:30,780 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:30,780 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:30,780 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,780 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:30,783 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,783 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:30,783 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:30,783 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:30,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:30,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:30,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:30,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:30,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:30,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:30,786 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:30,786 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:30,786 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:30,786 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:30,786 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:30,788 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:30,788 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:30,789 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:30,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:30,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:30,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:30,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:30,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:30,789 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:30,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:30,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:30,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:30,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:30,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:30,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:30,790 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:30,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:30,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,793 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,793 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:30,793 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:30,793 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:30,793 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:30,793 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:30,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:30,793 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:30,793 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:30,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:30,793 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,793 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:30,793 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:30,793 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:30,793 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:30,793 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:30,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:30,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:30,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:30,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 416 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042130820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,894 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:30,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:30,896 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:30,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:30,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:30,897 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:30,897 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:30,897 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:30,897 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:30,897 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:30,898 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:30,899 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,899 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:30,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:30,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:30,899 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,899 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:30,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:30,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:30,900 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:30,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:30,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:30,901 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:30,901 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:30,901 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:30,901 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:30,901 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:30,901 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:30,901 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:30,901 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:30,901 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:30,901 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:30,901 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:30,901 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,901 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:30,901 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:30,906 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,907 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:30,907 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:30,907 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:30,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:30,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44f99193[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:30,908 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:30,908 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:30,910 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:30,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:30,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:30,910 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:30,910 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,910 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:30,910 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:30,910 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:30,911 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,911 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:30,912 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:30,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:30,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:30,912 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:30,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:30,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:30,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:30,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:30,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:30,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:30,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:30,914 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:30,914 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:30,914 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:30,914 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:30,914 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:30,915 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:30,915 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:30,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:30,915 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:30,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:30,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:30,916 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:30,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:30,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:30,917 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:30,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:30,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:30,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 418 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042130924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,928 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,928 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:30,928 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:30,928 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:30,928 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:30,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:30,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:30,928 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:30,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:30,928 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:30,929 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:30,928 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:30,929 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:30,929 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:30,929 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:30,929 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:30,929 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:30,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:30,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:31,029 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:31,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:31,031 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:31,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:31,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:31,032 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:31,032 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:31,032 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:31,032 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:31,033 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,033 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,034 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:31,034 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:31,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:31,034 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,034 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:31,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:31,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,035 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:31,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:31,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:31,036 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:31,036 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:31,036 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:31,044 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:31,044 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:31,044 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:31,044 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:31,044 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:31,045 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:31,045 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:31,045 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:31,045 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,045 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:31,045 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:31,047 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,047 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,047 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,047 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:31,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:31,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a236ae7[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:31,049 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:31,049 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,051 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,051 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:31,051 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:31,051 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:31,051 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,051 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:31,053 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,054 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:31,054 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:31,054 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:31,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:31,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:31,056 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:31,056 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:31,056 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:31,057 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:31,057 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:31,059 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,059 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,059 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:31,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,059 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:31,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:31,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,061 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,066 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,066 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:31,066 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:31,066 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:31,066 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:31,066 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,066 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,066 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:31,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:31,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:31,066 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,066 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,066 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,066 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:31,066 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:31,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:31,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:31,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:31,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 420 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042131128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,167 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:31,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:31,169 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:31,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:31,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:31,170 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:31,170 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:31,170 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:31,170 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:31,171 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,171 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,172 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:31,172 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:31,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:31,172 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,172 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:31,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:31,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,173 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:31,173 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:31,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:31,174 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:31,174 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:31,174 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:31,174 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:31,174 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:31,174 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:31,174 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:31,174 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:31,174 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:31,174 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:31,174 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:31,174 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,174 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:31,174 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:31,177 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,177 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,177 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,177 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:31,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:31,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65ff2685[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:31,179 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:31,179 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,180 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,180 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:31,180 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:31,180 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:31,180 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,180 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:31,181 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,181 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:31,181 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:31,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,181 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:31,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:31,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:31,184 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:31,184 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:31,184 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:31,184 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:31,184 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:31,187 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,187 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:31,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,187 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,188 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304214fac8fd2d2784be884e11ff0ebfe4c45 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304214fac8fd2d2784be884e11ff0ebfe4c45 2023-04-21 01:54:31,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:31,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:31,188 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,191 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,195 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:31,195 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:31,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:31,195 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:31,195 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:31,195 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:31,195 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:31,195 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,195 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,195 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,195 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,195 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,196 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:31,196 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:31,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:31,196 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:31,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:31,296 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:31,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:31,298 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:31,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:31,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:31,300 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:31,300 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:31,300 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:31,300 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:31,300 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,301 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,302 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:31,302 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:31,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:31,302 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,302 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:31,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:31,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:31,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:31,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:31,303 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:31,303 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:31,303 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:31,303 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:31,303 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:31,303 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:31,303 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:31,304 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:31,304 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:31,304 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:31,304 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:31,304 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,304 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:31,304 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:31,305 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,305 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,306 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,306 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:31,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:31,307 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:31,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33c21387[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:31,307 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,312 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,312 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:31,312 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:31,312 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:31,312 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,312 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:31,313 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,313 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:31,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,313 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:31,313 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:31,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:31,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:31,316 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:31,316 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:31,316 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:31,316 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:31,316 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:31,317 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,317 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,317 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:31,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,318 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:31,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:31,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,323 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,326 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,326 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,326 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,326 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:31,326 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,326 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:31,326 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:31,326 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:31,326 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:31,326 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,326 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,326 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,326 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:31,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:31,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:31,326 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:31,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:31,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:31,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:31,426 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:31,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:31,429 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:31,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:31,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:31,430 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:31,430 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:31,430 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:31,430 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:31,430 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,430 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,433 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:31,433 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:31,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:31,433 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,433 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:31,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:31,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 422 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042131432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:31,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:31,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:31,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:31,435 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:31,435 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:31,435 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:31,435 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:31,435 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:31,435 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:31,436 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:31,436 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:31,436 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:31,436 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:31,436 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:31,436 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,436 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:31,436 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:31,437 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,437 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,438 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,438 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:31,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:31,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b3f7965[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:31,439 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:31,439 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,440 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,440 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:31,440 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:31,440 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:31,440 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,440 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:31,442 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:31,442 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:31,442 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,442 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:31,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:31,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:31,444 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:31,444 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:31,444 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:31,444 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:31,444 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:31,446 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,446 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,446 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:31,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:31,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:31,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,447 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,448 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,450 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,450 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,450 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:31,450 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:31,450 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:31,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:31,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:31,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:31,450 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,450 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,450 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,451 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,451 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,450 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:31,451 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:31,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:31,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:31,451 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:31,451 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:31,551 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:31,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:31,553 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:31,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:31,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:31,555 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:31,555 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:31,555 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:31,555 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:31,556 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,556 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,557 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:31,557 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:31,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:31,558 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,558 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:31,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:31,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:31,559 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:31,559 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:31,559 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:31,559 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:31,559 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:31,559 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:31,559 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:31,559 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:31,559 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:31,559 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:31,559 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:31,559 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:31,559 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:31,559 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,559 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:31,559 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:31,561 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,561 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,561 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,561 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:31,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:31,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a7a58b7[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:31,563 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:31,563 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,564 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,565 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:31,565 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:31,565 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:31,565 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,565 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:31,566 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:31,566 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:31,566 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,566 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:31,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:31,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:31,569 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:31,569 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:31,569 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:31,569 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:31,569 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:31,570 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,570 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,570 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:31,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:31,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,571 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:31,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,572 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,575 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,575 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:31,575 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:31,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:31,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:31,575 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:31,575 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:31,575 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,575 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:31,575 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:31,575 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,575 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,575 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,576 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:31,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:31,576 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:31,576 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:31,594 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=622, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/b3e6b0b2ac434e3a8a116e2d1476fd40 2023-04-21 01:54:31,676 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:31,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:31,679 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:31,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:31,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:31,681 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:31,681 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:31,681 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:31,681 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:31,681 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,681 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,684 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:31,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:31,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:31,684 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,685 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,685 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:31,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:31,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:31,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:31,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:31,686 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:31,686 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:31,686 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:31,688 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:31,688 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:31,688 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:31,688 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:31,688 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:31,688 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:31,688 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:31,688 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:31,688 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,688 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:31,688 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:31,690 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,690 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,690 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,690 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:31,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16ad9c73[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:31,692 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:31,692 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,693 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:31,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:31,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:31,694 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,694 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:31,699 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:31,699 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,699 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:31,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,699 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:31,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:31,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:31,701 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:31,701 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:31,701 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:31,701 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:31,701 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:31,703 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,703 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,703 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:31,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:31,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:31,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,704 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,706 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,708 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,708 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,708 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,708 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,708 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,708 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:31,708 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,708 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:31,709 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:31,709 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:31,709 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:31,709 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:31,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:31,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:31,709 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:31,710 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:31,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:31,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:31,810 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:31,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:31,812 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:31,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:31,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:31,813 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:31,813 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:31,813 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:31,814 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:31,814 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,814 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,815 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:31,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:31,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:31,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,816 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:31,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:31,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:31,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:31,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:31,817 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:31,817 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:31,817 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:31,818 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:31,818 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:31,818 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:31,818 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:31,818 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:31,818 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:31,818 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:31,818 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:31,818 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,818 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:31,818 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:31,820 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,822 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,822 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,822 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:31,825 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:31,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3550eb0a[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:31,825 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,826 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,826 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:31,826 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:31,826 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:31,826 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,826 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:31,828 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:31,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:31,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:31,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:31,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:31,830 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:31,830 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:31,830 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:31,830 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:31,830 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:31,832 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,832 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,832 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:31,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:31,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:31,833 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,834 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,837 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,837 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,837 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,837 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,837 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,837 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,837 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:31,837 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:31,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:31,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:31,837 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:31,837 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:31,837 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:31,837 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:31,838 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:31,838 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:31,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:31,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:31,938 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:31,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:31,940 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:31,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:31,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:31,941 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:31,941 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:31,941 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:31,941 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:31,941 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,942 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,943 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,943 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:31,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:31,943 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,943 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:31,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,944 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:31,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:31,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 424 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042131938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:31,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:31,944 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:31,944 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:31,945 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:31,945 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:31,945 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:31,946 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:31,946 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:31,946 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:31,946 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:31,946 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:31,946 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:31,946 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,946 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:31,946 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:31,947 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,947 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,948 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,948 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:31,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75645894[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:31,949 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:31,949 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,951 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,951 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:31,951 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:31,951 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:31,951 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,951 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:31,953 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,956 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,956 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:31,956 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:31,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:31,958 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:31,959 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:31,959 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:31,959 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:31,959 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:31,960 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,960 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:31,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:31,960 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:31,961 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:31,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:31,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:31,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,962 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:31,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:31,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:31,964 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,964 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:31,964 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:31,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:31,964 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:31,964 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:31,964 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:31,964 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:31,964 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:31,964 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:31,964 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:31,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:31,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:31,964 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:31,964 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:31,965 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:31,965 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:32,013 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=622 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/fa52ff9ca2194724b8047383b182f7ad 2023-04-21 01:54:32,023 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=622 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/7e23253b73f54b54985dacbbfcdda605 2023-04-21 01:54:32,028 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/b3e6b0b2ac434e3a8a116e2d1476fd40 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/b3e6b0b2ac434e3a8a116e2d1476fd40 2023-04-21 01:54:32,033 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/b3e6b0b2ac434e3a8a116e2d1476fd40, entries=150, sequenceid=622, filesize=25.4 K 2023-04-21 01:54:32,034 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/fa52ff9ca2194724b8047383b182f7ad as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/fa52ff9ca2194724b8047383b182f7ad 2023-04-21 01:54:32,038 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/fa52ff9ca2194724b8047383b182f7ad, entries=150, sequenceid=622, filesize=11.9 K 2023-04-21 01:54:32,039 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/7e23253b73f54b54985dacbbfcdda605 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/7e23253b73f54b54985dacbbfcdda605 2023-04-21 01:54:32,043 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/7e23253b73f54b54985dacbbfcdda605, entries=150, sequenceid=622, filesize=11.9 K 2023-04-21 01:54:32,045 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~192.98 KB/197616, currentSize=107.34 KB/109920 for e00828c94a25954f7bdc6de3ac3e2662 in 1271ms, sequenceid=622, compaction requested=true 2023-04-21 01:54:32,045 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:32,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:32,045 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:54:32,045 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:54:32,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:32,046 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:32,047 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 97546 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:54:32,047 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): e00828c94a25954f7bdc6de3ac3e2662/A is initiating minor compaction (all files) 2023-04-21 01:54:32,047 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e00828c94a25954f7bdc6de3ac3e2662/A in TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:32,047 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/564a5beb388b4e49a2cda44823c00ae6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/41a3de37d1aa47759fba629083b008a8, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/84b9ef0e620640e68de465b83f6a907f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/b3e6b0b2ac434e3a8a116e2d1476fd40] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp, totalSize=95.3 K 2023-04-21 01:54:32,047 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46710 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:54:32,047 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 564a5beb388b4e49a2cda44823c00ae6, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1682042066963 2023-04-21 01:54:32,047 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): e00828c94a25954f7bdc6de3ac3e2662/B is initiating minor compaction (all files) 2023-04-21 01:54:32,048 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e00828c94a25954f7bdc6de3ac3e2662/B in TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:32,048 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/0ccd49879c3b4f44a069de4fbd971a96, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/fc00e37fa6234c94bf0d82cf5e2bc213, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/feae6c9c5f8248938aa23dd5ae7804ec, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/fa52ff9ca2194724b8047383b182f7ad] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp, totalSize=45.6 K 2023-04-21 01:54:32,048 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 41a3de37d1aa47759fba629083b008a8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682042067685 2023-04-21 01:54:32,048 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 0ccd49879c3b4f44a069de4fbd971a96, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1682042066963 2023-04-21 01:54:32,048 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 84b9ef0e620640e68de465b83f6a907f, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1682042069851 2023-04-21 01:54:32,049 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting fc00e37fa6234c94bf0d82cf5e2bc213, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682042067685 2023-04-21 01:54:32,049 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting b3e6b0b2ac434e3a8a116e2d1476fd40, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=622, earliestPutTs=1682042070759 2023-04-21 01:54:32,049 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting feae6c9c5f8248938aa23dd5ae7804ec, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1682042069851 2023-04-21 01:54:32,049 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting fa52ff9ca2194724b8047383b182f7ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=622, earliestPutTs=1682042070759 2023-04-21 01:54:32,061 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): e00828c94a25954f7bdc6de3ac3e2662#B#compaction#191 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:32,063 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): e00828c94a25954f7bdc6de3ac3e2662#A#compaction#190 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:32,065 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:32,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:32,067 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:32,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:32,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:32,079 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:32,079 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:32,079 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:32,079 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:32,080 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,080 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,081 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:32,081 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:32,082 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,082 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:32,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:32,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:32,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:32,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:32,084 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:32,084 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:32,084 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:32,086 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/88198a0899c34bd280e459a8adf73720 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/88198a0899c34bd280e459a8adf73720 2023-04-21 01:54:32,086 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:32,087 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:32,087 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:32,087 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:32,087 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e00828c94a25954f7bdc6de3ac3e2662 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-21 01:54:32,087 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=A 2023-04-21 01:54:32,087 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:32,087 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=B 2023-04-21 01:54:32,087 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:32,087 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=C 2023-04-21 01:54:32,087 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:32,091 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e00828c94a25954f7bdc6de3ac3e2662/B of e00828c94a25954f7bdc6de3ac3e2662 into 88198a0899c34bd280e459a8adf73720(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:32,091 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:32,091 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., storeName=e00828c94a25954f7bdc6de3ac3e2662/B, priority=12, startTime=1682042072045; duration=0sec 2023-04-21 01:54:32,091 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:32,091 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:32,092 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34558 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:32,093 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): e00828c94a25954f7bdc6de3ac3e2662/C is initiating minor compaction (all files) 2023-04-21 01:54:32,093 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e00828c94a25954f7bdc6de3ac3e2662/C in TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:32,093 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/59c5fe3771794beb829e7d66b93175f7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/7f8ada7d5bd748eaa5ca3675217885ec, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/7e23253b73f54b54985dacbbfcdda605] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp, totalSize=33.7 K 2023-04-21 01:54:32,093 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 59c5fe3771794beb829e7d66b93175f7, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682042067685 2023-04-21 01:54:32,094 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 7f8ada7d5bd748eaa5ca3675217885ec, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1682042069851 2023-04-21 01:54:32,095 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 7e23253b73f54b54985dacbbfcdda605, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=622, earliestPutTs=1682042070759 2023-04-21 01:54:32,101 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): e00828c94a25954f7bdc6de3ac3e2662#C#compaction#192 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:32,128 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042124bf3829fd1d4191bfc8541a5fa8b8b6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042124bf3829fd1d4191bfc8541a5fa8b8b6 2023-04-21 01:54:32,129 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/8cd692f968a542938ab76574c308927c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/8cd692f968a542938ab76574c308927c 2023-04-21 01:54:32,142 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e00828c94a25954f7bdc6de3ac3e2662/C of e00828c94a25954f7bdc6de3ac3e2662 into 8cd692f968a542938ab76574c308927c(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:32,144 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:32,144 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., storeName=e00828c94a25954f7bdc6de3ac3e2662/C, priority=13, startTime=1682042072046; duration=0sec 2023-04-21 01:54:32,144 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:32,152 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=641, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/b2770d97d2214656ad05fa01351a20a9 2023-04-21 01:54:32,178 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/8da3d20fc464415e97607af9706d641d 2023-04-21 01:54:32,201 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/5f22e79250734f55ab836d2c36ffd9fd 2023-04-21 01:54:32,207 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/b2770d97d2214656ad05fa01351a20a9 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/b2770d97d2214656ad05fa01351a20a9 2023-04-21 01:54:32,211 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/b2770d97d2214656ad05fa01351a20a9, entries=150, sequenceid=641, filesize=25.4 K 2023-04-21 01:54:32,212 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/8da3d20fc464415e97607af9706d641d as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/8da3d20fc464415e97607af9706d641d 2023-04-21 01:54:32,216 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/8da3d20fc464415e97607af9706d641d, entries=150, sequenceid=641, filesize=11.9 K 2023-04-21 01:54:32,217 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/5f22e79250734f55ab836d2c36ffd9fd as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/5f22e79250734f55ab836d2c36ffd9fd 2023-04-21 01:54:32,221 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/5f22e79250734f55ab836d2c36ffd9fd, entries=150, sequenceid=641, filesize=11.9 K 2023-04-21 01:54:32,222 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for e00828c94a25954f7bdc6de3ac3e2662 in 135ms, sequenceid=641, compaction requested=false 2023-04-21 01:54:32,222 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:32,222 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:32,222 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:32,222 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:32,222 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,224 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:32,224 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:32,230 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,230 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:32,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:32,230 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,230 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:32,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:32,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:32,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:32,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:32,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d03096f[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:32,233 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:32,233 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,234 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,234 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:32,234 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:32,234 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:32,234 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,234 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:32,235 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:32,235 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:32,235 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:32,235 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:32,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:32,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:32,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:32,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:32,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:32,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:32,239 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:32,239 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:32,239 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:32,239 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:32,239 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:32,241 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,241 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:32,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:32,241 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:32,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:32,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:32,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:32,242 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:32,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,243 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:32,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,248 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,249 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:32,249 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:32,249 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:32,249 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:32,249 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,249 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:32,249 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:32,249 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:32,249 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:32,249 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:32,250 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,250 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,250 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,349 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:32,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:32,352 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:32,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:32,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:32,353 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:32,353 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:32,353 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:32,353 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:32,354 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,354 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,355 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,355 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:32,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:32,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:32,356 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,356 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:32,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:32,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:32,360 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:32,360 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:32,360 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:32,360 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:32,360 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:32,360 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:32,360 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:32,360 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:32,360 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:32,361 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:32,361 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:32,361 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:32,361 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:32,361 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,361 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:32,361 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:32,362 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,362 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:32,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:32,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:32,363 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,363 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:32,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:32,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:32,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:32,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@573756b8[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:32,365 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:32,365 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,366 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,366 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:32,366 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:32,366 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:32,366 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,366 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:32,369 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,369 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:32,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:32,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:32,369 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:32,369 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:32,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:32,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:32,372 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:32,372 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:32,372 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:32,372 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:32,372 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:32,374 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,374 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:32,374 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:32,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,374 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:32,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:32,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,376 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,378 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:32,379 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,379 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:32,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:32,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:32,379 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,379 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,379 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:32,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:32,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:32,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:32,379 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:32,379 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:32,379 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,379 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,379 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,379 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,379 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:32,380 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:32,480 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:32,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:32,482 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:32,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:32,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:32,483 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:32,483 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:32,484 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:32,484 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:32,484 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,484 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,485 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,486 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,486 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:32,486 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:32,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:32,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:32,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:32,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:32,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:32,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:32,496 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:32,496 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:32,496 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:32,496 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:32,497 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:32,497 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:32,497 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:32,497 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:32,498 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:32,498 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:32,498 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:32,498 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,498 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:32,498 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:32,500 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,500 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:32,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:32,501 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,501 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:32,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:32,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:32,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:32,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:32,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@650dffef[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:32,503 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:32,503 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,504 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,504 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:32,505 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:32,505 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:32,505 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,505 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:32,506 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,506 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:32,506 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:32,506 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:32,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:32,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:32,509 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:32,509 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:32,509 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:32,509 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:32,509 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:32,511 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,511 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:32,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,511 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:32,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:32,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:32,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:32,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:32,512 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:32,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,513 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:32,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,516 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:32,516 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:32,516 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,517 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,517 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,517 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,517 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,517 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,517 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,517 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:32,517 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:32,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:32,517 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:32,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:32,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:32,519 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:32,519 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:32,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:32,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:32,619 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:32,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:32,621 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:32,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:32,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:32,623 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:32,623 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:32,623 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:32,623 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:32,624 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,624 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,625 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:32,625 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:32,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:32,626 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,626 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:32,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:32,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:32,627 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:32,627 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:32,627 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:32,627 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:32,627 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:32,628 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:32,628 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:32,628 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:32,628 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:32,628 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:32,628 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:32,628 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:32,628 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:32,628 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,628 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:32,628 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:32,630 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,630 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:32,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:32,630 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,630 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:32,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:32,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:32,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:32,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:32,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32fabb22[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:32,632 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:32,632 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,634 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,634 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:32,634 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:32,634 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:32,634 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,634 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:32,637 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:32,637 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:32,637 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:32,637 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:32,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:32,640 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:32,640 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:32,640 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:32,640 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:32,640 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:32,642 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,642 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:32,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:32,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,642 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:32,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:32,642 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:32,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:32,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:32,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,644 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:32,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,649 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:32,649 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:32,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:32,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:32,650 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,650 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,650 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,650 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,650 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,650 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:32,650 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,650 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:32,650 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:32,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:32,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:32,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:32,650 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:32,650 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:32,650 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,750 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:32,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:32,753 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:32,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:32,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:32,755 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:32,755 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:32,755 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:32,755 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:32,756 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,756 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,789 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:32,789 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:32,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:32,790 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,790 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:32,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:32,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,791 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:32,791 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:32,792 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:32,792 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:32,792 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:32,796 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:32,796 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:32,796 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:32,796 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:32,796 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:32,796 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:32,796 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:32,796 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:32,796 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,796 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:32,796 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:32,799 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,799 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:32,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:32,799 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,799 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:32,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:32,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:32,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:32,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:32,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7079463[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:32,801 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:32,801 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,802 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,802 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:32,802 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:32,802 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:32,803 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,803 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:32,804 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:32,804 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:32,804 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:32,805 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:32,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:32,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:32,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:32,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:32,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:32,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:32,808 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:32,808 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:32,808 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:32,808 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:32,808 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:32,811 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,811 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:32,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:32,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,811 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:32,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:32,811 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:32,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:32,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:32,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,812 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,815 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:32,815 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,815 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:32,815 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,815 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:32,815 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:32,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:32,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:32,815 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:32,815 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,815 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,815 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,815 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,815 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,815 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:32,815 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:32,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:32,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:32,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:32,916 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:32,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:32,918 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:32,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:32,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:32,936 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:32,936 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:32,936 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:32,936 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:32,937 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,937 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,938 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:32,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:32,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:32,938 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:32,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:32,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,939 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:32,940 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:32,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:32,940 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:32,940 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:32,940 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:32,944 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:32,944 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:32,944 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:32,944 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:32,945 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:32,945 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:32,945 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:32,945 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:32,945 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,945 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:32,945 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:32,947 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,947 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:32,947 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,947 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:32,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:32,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ae880ad[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:32,949 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:32,949 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,951 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,951 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:32,951 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:32,951 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:32,951 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,952 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:32,953 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,953 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:32,953 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:32,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:32,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:32,953 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:32,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:32,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:32,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:32,956 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:32,956 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:32,956 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:32,956 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:32,956 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:32,957 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,957 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:32,957 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:32,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:32,958 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:32,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,960 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,963 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:32,963 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:32,963 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:32,963 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,963 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,963 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:32,963 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:32,963 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,963 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:32,963 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:32,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:32,963 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:32,963 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:32,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:32,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:32,964 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:32,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:32,964 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:33,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:33,021 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e00828c94a25954f7bdc6de3ac3e2662 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-21 01:54:33,021 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=A 2023-04-21 01:54:33,021 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:33,021 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=B 2023-04-21 01:54:33,021 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:33,021 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=C 2023-04-21 01:54:33,022 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:33,064 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:33,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:33,067 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:33,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:33,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:33,069 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:33,069 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:33,069 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:33,069 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:33,069 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,069 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:33,071 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:33,071 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:33,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:33,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:33,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,072 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,072 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:33,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:33,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:33,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:33,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:33,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:33,080 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:33,080 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:33,080 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:33,088 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:33,088 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:33,088 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:33,088 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:33,090 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:33,091 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:33,091 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,091 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:33,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682042073067, max timestamp=1682042073089Num uniques 859; 2023-04-21 01:54:33,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:54:33,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682042073067, max timestamp=1682042073089Num uniques 859; 2023-04-21 01:54:33,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:54:33,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:33,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:33,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6090f898[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:33,094 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:33,094 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042073067, max timestamp=1682042073089Num uniques 859; 2023-04-21 01:54:33,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:54:33,095 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,095 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:33,095 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:33,095 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:33,096 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,096 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:33,097 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:33,097 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,097 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:33,097 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:33,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:33,101 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:33,101 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:33,101 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:33,101 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:33,101 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:33,105 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,105 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,105 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:33,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:33,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:33,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:33,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:33,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,107 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,108 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,112 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,112 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:33,112 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:33,112 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:33,112 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:33,112 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:33,112 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:33,112 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:33,112 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:33,112 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:33,112 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,112 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,112 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,112 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:33,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:33,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:33,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:33,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 466 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042133158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,212 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:33,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:33,215 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:33,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:33,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:33,216 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:33,216 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:33,216 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:33,216 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:33,217 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,217 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:33,225 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:33,225 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:33,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:33,225 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,225 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:33,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:33,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:33,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:33,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:33,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:33,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:33,226 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:33,226 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:33,226 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:33,226 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:33,226 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:33,227 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:33,227 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:33,227 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:33,227 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:33,227 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:33,227 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:33,227 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,227 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:33,227 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:33,230 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,230 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:33,230 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,230 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:33,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:33,231 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:33,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d46c03[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:33,231 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,232 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,233 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:33,233 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:33,233 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:33,233 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,233 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:33,235 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,235 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:33,235 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:33,235 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:33,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:33,237 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:33,237 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:33,237 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:33,237 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:33,237 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:33,239 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,239 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,239 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:33,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:33,239 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,241 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:33,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,244 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:33,245 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:33,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:33,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:33,245 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,245 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:33,245 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:33,245 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:33,245 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:33,245 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:33,245 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,245 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,245 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,245 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,245 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:33,245 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:33,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:33,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:33,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:33,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 468 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042133262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:33,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57888 deadline: 1682042133301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,307 DEBUG [Thread-1022] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8169 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:54:33,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:33,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57878 deadline: 1682042133307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,311 DEBUG [Thread-1016] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8175 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:54:33,330 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/c9b0cbf8c91c437186da477445afce09 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/c9b0cbf8c91c437186da477445afce09 2023-04-21 01:54:33,335 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e00828c94a25954f7bdc6de3ac3e2662/A of e00828c94a25954f7bdc6de3ac3e2662 into c9b0cbf8c91c437186da477445afce09(size=26.0 K), total size for store is 51.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-21 01:54:33,335 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:33,335 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., storeName=e00828c94a25954f7bdc6de3ac3e2662/A, priority=12, startTime=1682042072045; duration=1sec 2023-04-21 01:54:33,335 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:33,345 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:33,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:33,347 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:33,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:33,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:33,349 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:33,349 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:33,349 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:33,349 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:33,349 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,349 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:33,350 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:33,350 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:33,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:33,351 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:33,351 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:33,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:33,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:33,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:33,353 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:33,353 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:33,353 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:33,353 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:33,353 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:33,353 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:33,353 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:33,353 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:33,353 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:33,353 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:33,353 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:33,353 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:33,354 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:33,354 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,354 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:33,354 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:33,356 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,356 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:33,356 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,356 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:33,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:33,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c4b5416[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:33,357 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:33,358 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,359 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,359 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:33,359 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:33,359 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:33,359 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,359 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:33,361 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,361 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:33,361 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:33,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:33,361 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:33,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:33,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:33,364 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:33,364 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:33,364 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:33,364 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:33,364 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:33,366 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,366 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:33,366 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:33,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:33,367 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,368 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,370 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:33,370 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:33,371 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:33,370 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:33,371 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:33,370 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,371 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:33,371 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:33,371 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:33,371 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,371 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,371 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:33,371 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:33,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:33,371 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:33,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:33,449 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421023616cbfc3d49eab843ef01fe6171f4 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421023616cbfc3d49eab843ef01fe6171f4 2023-04-21 01:54:33,460 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=658, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/6ec4139669b74983a03cd68e61acc74a 2023-04-21 01:54:33,471 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:33,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:33,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:33,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 470 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042133469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,474 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:33,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:33,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:33,480 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:33,480 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:33,480 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:33,480 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:33,480 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,481 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:33,482 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:33,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:33,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:33,482 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:33,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,483 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,483 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:33,484 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=658 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/674ec3b4f40144ba89573c82c7327439 2023-04-21 01:54:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:33,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:33,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:33,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:33,491 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:33,491 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:33,491 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:33,500 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:33,500 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:33,500 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:33,500 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:33,500 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:33,500 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:33,500 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:33,500 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:33,500 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,500 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:33,500 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:33,503 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,503 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:33,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:33,503 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,503 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:33,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:33,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:33,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:33,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:33,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@128846da[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:33,505 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:33,505 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,507 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,507 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:33,507 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:33,507 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:33,508 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,508 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:33,509 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:33,509 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:33,509 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:33,509 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:33,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:33,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:33,512 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=658 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/31c7f8abbc564582b921d56c6a195c43 2023-04-21 01:54:33,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:33,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:33,513 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:33,513 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:33,513 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:33,513 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:33,514 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:33,517 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/6ec4139669b74983a03cd68e61acc74a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/6ec4139669b74983a03cd68e61acc74a 2023-04-21 01:54:33,517 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,517 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:33,517 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:33,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:33,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:33,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:33,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:33,519 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:33,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,521 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:33,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:33,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,523 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/6ec4139669b74983a03cd68e61acc74a, entries=350, sequenceid=658, filesize=52.9 K 2023-04-21 01:54:33,524 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/674ec3b4f40144ba89573c82c7327439 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/674ec3b4f40144ba89573c82c7327439 2023-04-21 01:54:33,525 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,525 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:33,525 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:33,526 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,526 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:33,526 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:33,526 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:33,526 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:33,525 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:33,526 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:33,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:33,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:33,526 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,526 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:33,526 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,526 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:33,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:33,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:33,528 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/674ec3b4f40144ba89573c82c7327439, entries=150, sequenceid=658, filesize=11.9 K 2023-04-21 01:54:33,529 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/31c7f8abbc564582b921d56c6a195c43 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/31c7f8abbc564582b921d56c6a195c43 2023-04-21 01:54:33,536 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/31c7f8abbc564582b921d56c6a195c43, entries=150, sequenceid=658, filesize=11.9 K 2023-04-21 01:54:33,537 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.94 KB/135110 for e00828c94a25954f7bdc6de3ac3e2662 in 516ms, sequenceid=658, compaction requested=true 2023-04-21 01:54:33,537 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:33,537 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:33,537 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:33,537 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-21 01:54:33,537 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:33,537 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:33,539 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 106852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:33,539 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:33,539 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): e00828c94a25954f7bdc6de3ac3e2662/A is initiating minor compaction (all files) 2023-04-21 01:54:33,539 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): e00828c94a25954f7bdc6de3ac3e2662/B is initiating minor compaction (all files) 2023-04-21 01:54:33,539 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e00828c94a25954f7bdc6de3ac3e2662/A in TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:33,539 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e00828c94a25954f7bdc6de3ac3e2662/B in TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:33,539 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/c9b0cbf8c91c437186da477445afce09, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/b2770d97d2214656ad05fa01351a20a9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/6ec4139669b74983a03cd68e61acc74a] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp, totalSize=104.3 K 2023-04-21 01:54:33,539 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/88198a0899c34bd280e459a8adf73720, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/8da3d20fc464415e97607af9706d641d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/674ec3b4f40144ba89573c82c7327439] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp, totalSize=36.2 K 2023-04-21 01:54:33,539 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting c9b0cbf8c91c437186da477445afce09, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=622, earliestPutTs=1682042070759 2023-04-21 01:54:33,540 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 88198a0899c34bd280e459a8adf73720, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=622, earliestPutTs=1682042070759 2023-04-21 01:54:33,540 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting b2770d97d2214656ad05fa01351a20a9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1682042070800 2023-04-21 01:54:33,540 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 6ec4139669b74983a03cd68e61acc74a, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1682042072974 2023-04-21 01:54:33,541 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 8da3d20fc464415e97607af9706d641d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1682042070800 2023-04-21 01:54:33,542 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 674ec3b4f40144ba89573c82c7327439, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1682042073006 2023-04-21 01:54:33,569 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): e00828c94a25954f7bdc6de3ac3e2662#B#compaction#200 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:33,594 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): e00828c94a25954f7bdc6de3ac3e2662#A#compaction#199 average throughput is 0.58 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:33,626 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:33,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:33,628 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:33,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:33,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:33,632 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:33,632 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:33,632 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:33,632 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:33,632 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,633 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:33,634 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:33,634 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:33,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:33,634 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,634 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:33,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:33,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:33,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:33,635 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:33,635 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:33,635 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:33,635 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:33,635 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:33,635 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:33,635 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:33,635 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:33,635 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:33,636 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e00828c94a25954f7bdc6de3ac3e2662 3/3 column families, dataSize=131.94 KB heapSize=375.22 KB 2023-04-21 01:54:33,636 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=A 2023-04-21 01:54:33,636 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:33,636 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=B 2023-04-21 01:54:33,636 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:33,636 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=C 2023-04-21 01:54:33,636 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:33,659 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042173a106fda05145ba8dc46c13145b46ca to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042173a106fda05145ba8dc46c13145b46ca 2023-04-21 01:54:33,666 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=695, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/ac02532900614daba46fd4f65c055b8a 2023-04-21 01:54:33,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:33,780 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:33,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:33,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 479 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042133800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:33,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:33,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 481 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042133901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:34,010 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/7e1ae28fdada4fb0b97f542bf0253c29 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/7e1ae28fdada4fb0b97f542bf0253c29 2023-04-21 01:54:34,015 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e00828c94a25954f7bdc6de3ac3e2662/B of e00828c94a25954f7bdc6de3ac3e2662 into 7e1ae28fdada4fb0b97f542bf0253c29(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:34,015 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:34,015 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., storeName=e00828c94a25954f7bdc6de3ac3e2662/B, priority=13, startTime=1682042073537; duration=0sec 2023-04-21 01:54:34,015 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:34,015 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:34,016 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:34,016 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): e00828c94a25954f7bdc6de3ac3e2662/C is initiating minor compaction (all files) 2023-04-21 01:54:34,016 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e00828c94a25954f7bdc6de3ac3e2662/C in TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:34,016 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/8cd692f968a542938ab76574c308927c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/5f22e79250734f55ab836d2c36ffd9fd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/31c7f8abbc564582b921d56c6a195c43] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp, totalSize=36.2 K 2023-04-21 01:54:34,017 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 8cd692f968a542938ab76574c308927c, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=622, earliestPutTs=1682042070759 2023-04-21 01:54:34,017 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 5f22e79250734f55ab836d2c36ffd9fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1682042070800 2023-04-21 01:54:34,018 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 31c7f8abbc564582b921d56c6a195c43, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1682042073006 2023-04-21 01:54:34,025 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): e00828c94a25954f7bdc6de3ac3e2662#C#compaction#203 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:34,079 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=695 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/bed30adb0e9c4378a4fd63be8c5ac17e 2023-04-21 01:54:34,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:34,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 483 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042134106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:34,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:34,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 485 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042134411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:34,439 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/f2572b68c8844941a36120c5530c77bb as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/f2572b68c8844941a36120c5530c77bb 2023-04-21 01:54:34,443 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e00828c94a25954f7bdc6de3ac3e2662/C of e00828c94a25954f7bdc6de3ac3e2662 into f2572b68c8844941a36120c5530c77bb(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:34,444 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:34,444 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., storeName=e00828c94a25954f7bdc6de3ac3e2662/C, priority=13, startTime=1682042073537; duration=0sec 2023-04-21 01:54:34,444 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:34,448 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/a136d8b7b88545a68d8c4e3b976772dd as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/a136d8b7b88545a68d8c4e3b976772dd 2023-04-21 01:54:34,454 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e00828c94a25954f7bdc6de3ac3e2662/A of e00828c94a25954f7bdc6de3ac3e2662 into a136d8b7b88545a68d8c4e3b976772dd(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:34,454 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:34,454 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., storeName=e00828c94a25954f7bdc6de3ac3e2662/A, priority=13, startTime=1682042073537; duration=0sec 2023-04-21 01:54:34,454 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:34,493 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=695 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/dd7dea90249f407b84f0c920ab0a6310 2023-04-21 01:54:34,500 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/ac02532900614daba46fd4f65c055b8a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/ac02532900614daba46fd4f65c055b8a 2023-04-21 01:54:34,505 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/ac02532900614daba46fd4f65c055b8a, entries=150, sequenceid=695, filesize=25.4 K 2023-04-21 01:54:34,506 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/bed30adb0e9c4378a4fd63be8c5ac17e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/bed30adb0e9c4378a4fd63be8c5ac17e 2023-04-21 01:54:34,512 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/bed30adb0e9c4378a4fd63be8c5ac17e, entries=150, sequenceid=695, filesize=11.9 K 2023-04-21 01:54:34,513 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/dd7dea90249f407b84f0c920ab0a6310 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/dd7dea90249f407b84f0c920ab0a6310 2023-04-21 01:54:34,517 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/dd7dea90249f407b84f0c920ab0a6310, entries=150, sequenceid=695, filesize=11.9 K 2023-04-21 01:54:34,518 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~375.17 KB/384176, currentSize=46.96 KB/48090 for e00828c94a25954f7bdc6de3ac3e2662 in 883ms, sequenceid=695, compaction requested=false 2023-04-21 01:54:34,518 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:34,518 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:34,518 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:34,518 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:34,518 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:34,518 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:34,518 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:34,519 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:34,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:34,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:34,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:34,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:34,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:34,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:34,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:34,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:34,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:34,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:34,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:34,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:34,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75bb382b[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:34,523 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:34,523 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:34,528 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:34,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:34,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:34,528 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:34,529 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:34,529 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:34,530 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:34,530 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:34,530 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:34,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:34,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:34,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:34,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:34,535 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:34,535 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:34,535 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:34,535 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:34,535 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:34,537 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:34,537 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:34,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:34,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:34,537 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:34,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:34,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:34,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:34,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:34,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:34,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:34,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:34,538 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:34,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:34,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:34,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:34,541 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:34,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:34,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:34,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:34,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:34,543 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:34,543 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:34,543 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:34,543 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:34,544 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:34,543 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:34,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:34,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:34,544 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:34,544 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:34,544 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:34,544 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:34,544 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:34,544 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:34,544 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:34,544 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:34,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:34,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:34,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:34,644 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:34,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:34,646 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:34,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:34,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:34,647 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:34,647 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:34,648 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:34,648 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:34,648 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:34,648 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:34,651 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:34,651 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:34,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:34,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:34,651 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:34,651 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:34,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:34,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:34,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:34,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:34,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:34,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:34,653 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:34,653 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:34,653 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:34,653 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:34,653 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:34,653 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:34,653 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:34,653 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:34,653 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e00828c94a25954f7bdc6de3ac3e2662 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-21 01:54:34,653 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=A 2023-04-21 01:54:34,653 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:34,653 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=B 2023-04-21 01:54:34,654 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:34,654 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=C 2023-04-21 01:54:34,654 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:34,670 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042107a7fe2e269a44069be058657c4cb3d1 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042107a7fe2e269a44069be058657c4cb3d1 2023-04-21 01:54:34,675 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=708, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/23de29cfc19b49fd87bc68ee2a853fb1 2023-04-21 01:54:34,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:34,915 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:34,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682042074926, max timestamp=1682042074949Num uniques 859; 2023-04-21 01:54:34,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:54:34,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682042074926, max timestamp=1682042074949Num uniques 859; 2023-04-21 01:54:34,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682042074926, max timestamp=1682042074949Num uniques 859; 2023-04-21 01:54:34,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:54:34,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:54:34,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:34,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 520 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042134987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,086 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=708 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/45d25955ed134c50a1d4fab00c2aaaa4 2023-04-21 01:54:35,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:35,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 522 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042135089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,292 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:35,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 524 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042135291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,499 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=708 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/e16255e566eb41a49d9d609529caa83d 2023-04-21 01:54:35,503 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/23de29cfc19b49fd87bc68ee2a853fb1 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/23de29cfc19b49fd87bc68ee2a853fb1 2023-04-21 01:54:35,507 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/23de29cfc19b49fd87bc68ee2a853fb1, entries=150, sequenceid=708, filesize=25.4 K 2023-04-21 01:54:35,508 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/45d25955ed134c50a1d4fab00c2aaaa4 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/45d25955ed134c50a1d4fab00c2aaaa4 2023-04-21 01:54:35,512 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/45d25955ed134c50a1d4fab00c2aaaa4, entries=150, sequenceid=708, filesize=11.9 K 2023-04-21 01:54:35,513 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/e16255e566eb41a49d9d609529caa83d as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/e16255e566eb41a49d9d609529caa83d 2023-04-21 01:54:35,516 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/e16255e566eb41a49d9d609529caa83d, entries=150, sequenceid=708, filesize=11.9 K 2023-04-21 01:54:35,517 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e00828c94a25954f7bdc6de3ac3e2662 in 864ms, sequenceid=708, compaction requested=true 2023-04-21 01:54:35,517 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:35,517 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:35,517 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:35,517 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:35,517 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:35,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:35,521 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,521 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:35,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:35,522 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,522 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:35,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:35,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:35,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:35,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:35,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@512be6eb[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:35,524 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:35,524 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,525 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,525 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:35,525 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:35,526 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:35,526 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,526 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:35,527 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:35,527 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,527 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:35,527 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:35,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:35,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:35,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:35,530 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:35,530 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:35,530 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:35,530 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:35,530 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:35,534 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,534 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:35,534 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:35,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:35,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:35,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:35,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:35,535 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:35,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,536 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:35,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,539 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,539 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:35,539 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:35,539 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,539 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:35,539 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:35,539 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:35,539 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:35,539 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,539 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,539 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:35,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:35,539 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:35,539 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:35,539 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:35,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:35,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:35,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:35,599 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e00828c94a25954f7bdc6de3ac3e2662 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-21 01:54:35,599 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=A 2023-04-21 01:54:35,599 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:35,599 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=B 2023-04-21 01:54:35,599 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:35,599 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=C 2023-04-21 01:54:35,600 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:35,614 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304212bfc52f3bfdc46b980a554802969bc5e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304212bfc52f3bfdc46b980a554802969bc5e 2023-04-21 01:54:35,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:35,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 533 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042135612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,640 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:35,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:35,642 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:35,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:35,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:35,643 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:35,643 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:35,643 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:35,643 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:35,643 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,643 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:35,645 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:35,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:35,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,646 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:35,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:35,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:35,646 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,646 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:35,646 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:35,646 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:35,646 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:35,646 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:35,646 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:35,647 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:35,647 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:35,647 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:35,647 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:35,647 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:35,647 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:35,647 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:35,647 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:35,647 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,647 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:35,647 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:35,649 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,649 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:35,649 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,649 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:35,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:35,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:35,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3726f33c[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:35,651 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:35,651 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,652 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,653 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:35,653 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,653 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:35,653 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:35,653 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:35,654 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:35,654 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,654 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:35,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:35,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:35,654 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:35,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:35,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:35,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:35,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:35,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:35,656 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:35,656 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:35,656 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:35,656 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:35,656 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:35,658 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,658 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:35,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:35,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,658 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:35,658 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:35,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:35,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:35,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:35,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,660 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:35,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,662 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,662 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:35,662 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:35,662 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,662 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,662 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:35,662 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:35,662 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:35,663 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:35,662 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,663 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:35,663 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,663 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:35,663 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:35,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:35,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:35,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:35,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:35,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:35,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:35,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 535 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042135716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,763 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:35,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:35,765 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:35,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:35,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:35,766 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:35,766 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:35,766 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:35,766 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:35,767 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,767 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:35,769 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,769 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:35,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:35,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:35,770 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,770 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:35,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:35,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:35,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:35,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:35,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:35,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:35,771 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:35,771 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:35,771 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:35,771 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:35,772 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:35,772 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:35,772 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:35,772 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:35,772 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:35,772 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:35,772 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:35,772 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,772 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:35,772 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:35,774 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,774 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:35,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:35,774 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,774 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:35,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:35,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:35,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:35,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:35,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6748dd94[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:35,776 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:35,776 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,777 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,777 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:35,777 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:35,777 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,778 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:35,777 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:35,782 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,782 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:35,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:35,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:35,782 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:35,783 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:35,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:35,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:35,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:35,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:35,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:35,785 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:35,785 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:35,785 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:35,785 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:35,785 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:35,786 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,786 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:35,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:35,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,787 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:35,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:35,787 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:35,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:35,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,788 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:35,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,790 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,790 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:35,790 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:35,790 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,790 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,790 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,790 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:35,790 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:35,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:35,790 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:35,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:35,790 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:35,790 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:35,790 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:35,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:35,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:35,791 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:35,791 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:35,891 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:35,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:35,893 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:35,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:35,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:35,894 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:35,894 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:35,894 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:35,894 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:35,895 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,895 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:35,896 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:35,896 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:35,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:35,896 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,896 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:35,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:35,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:35,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:35,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:35,897 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:35,897 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:35,897 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:35,897 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:35,897 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:35,898 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:35,898 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:35,898 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:35,898 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:35,898 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:35,898 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:35,898 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:35,898 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:35,898 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,898 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:35,898 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:35,900 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,900 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:35,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:35,901 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:35,901 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:35,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:35,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:35,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:35,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b1edb36[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:35,920 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:35,920 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,921 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,921 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:35,921 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:35,921 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,921 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:35,921 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:35,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:35,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 537 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042135920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,922 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:35,922 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:35,922 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,922 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:35,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:35,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:35,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:35,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:35,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:35,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:35,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:35,925 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:35,925 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:35,925 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:35,925 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:35,925 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:35,926 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,926 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:35,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:35,926 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:35,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:35,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:35,927 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:35,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:35,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:35,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,929 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:35,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:35,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,933 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,933 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:35,933 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:35,933 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:35,933 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:35,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:35,933 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:35,933 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:35,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:35,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:35,933 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:35,933 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:35,933 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,933 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:35,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:35,933 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:35,933 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:35,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:35,933 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,020 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=745, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/681490a41f44431b87bcd14fefe3dee1 2023-04-21 01:54:36,033 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:36,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:36,035 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:36,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:36,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:36,037 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:36,037 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:36,037 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:36,037 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:36,037 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,037 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,038 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,038 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:36,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:36,038 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,038 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:36,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:36,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:36,040 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:36,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:36,040 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:36,040 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:36,040 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:36,040 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:36,040 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:36,040 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:36,040 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:36,040 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:36,040 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:36,040 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:36,040 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:36,040 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,040 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:36,040 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:36,042 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,042 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:36,042 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:36,042 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:36,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:36,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@458c6000[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:36,043 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:36,044 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,045 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,045 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:36,045 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:36,045 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:36,047 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:36,047 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:36,047 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:36,055 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,055 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,055 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:36,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:36,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:36,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:36,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:36,060 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:36,060 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:36,060 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:36,060 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:36,060 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:36,062 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,062 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:36,062 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:36,063 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:36,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:36,065 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:36,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,067 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,067 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,067 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,067 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,067 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:36,067 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:36,067 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:36,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:36,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:36,067 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:36,068 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:36,068 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:36,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:36,067 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:36,067 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,068 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,068 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:36,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:36,168 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:36,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:36,170 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:36,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:36,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:36,171 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:36,171 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:36,172 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:36,172 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:36,172 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,172 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,173 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:36,173 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:36,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:36,173 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,174 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:36,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:36,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:36,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:36,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:36,175 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:36,175 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:36,175 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:36,175 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:36,175 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:36,175 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:36,175 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:36,175 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:36,175 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:36,175 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:36,175 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:36,175 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,175 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:36,175 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:36,179 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,180 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,180 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,180 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:36,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:36,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:36,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:36,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:36,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:36,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:36,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4533c49e[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:36,183 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:36,183 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,184 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,184 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:36,184 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:36,184 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:36,184 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,184 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:36,185 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:36,185 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:36,185 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:36,185 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:36,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:36,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:36,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:36,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:36,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:36,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:36,188 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:36,188 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:36,188 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:36,188 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:36,188 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:36,189 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,189 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:36,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,189 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:36,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:36,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:36,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:36,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,190 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:36,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:36,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,194 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,199 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:36,199 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:36,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:36,199 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:36,199 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:36,199 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:36,199 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:36,199 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,199 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,199 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:36,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:36,199 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:36,199 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,200 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:36,200 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,200 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:36,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 539 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042136222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,300 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:36,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:36,302 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:36,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:36,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:36,303 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:36,303 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:36,303 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:36,303 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:36,304 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,304 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,305 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:36,305 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:36,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:36,305 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,305 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:36,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:36,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:36,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:36,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:36,306 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:36,306 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:36,306 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:36,307 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:36,307 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:36,307 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:36,307 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:36,307 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:36,307 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:36,307 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:36,307 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:36,307 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,307 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:36,307 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:36,308 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,308 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:36,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:36,309 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,309 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:36,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:36,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:36,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:36,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:36,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@643615d1[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:36,310 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:36,310 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,311 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,311 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:36,311 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:36,311 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:36,311 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,311 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:36,313 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:36,313 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,313 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:36,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:36,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:36,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:36,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:36,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:36,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:36,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:36,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:36,316 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:36,316 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:36,317 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:36,317 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:36,317 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:36,318 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,318 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:36,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:36,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,318 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:36,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:36,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:36,319 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:36,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:36,320 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:36,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,323 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:36,323 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:36,323 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,323 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,323 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,323 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,323 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,323 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,323 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:36,323 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:36,323 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:36,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:36,323 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:36,323 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:36,324 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:36,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:36,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:36,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:36,424 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:36,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:36,426 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:36,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:36,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:36,427 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:36,427 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:36,427 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:36,427 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:36,427 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,427 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,428 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:36,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:36,428 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:36,429 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,429 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:36,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:36,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:36,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:36,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:36,430 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:36,430 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:36,430 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:36,430 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:36,431 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:36,431 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:36,431 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:36,431 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:36,431 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:36,431 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:36,431 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:36,431 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=745 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/c03a59cbce6a4bd48f6c976fba897e34 2023-04-21 01:54:36,431 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,431 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:36,432 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:36,435 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,435 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:36,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:36,435 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,435 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:36,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:36,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:36,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59cca50a[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:36,441 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:36,441 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,442 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,443 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,443 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:36,443 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:36,444 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:36,444 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:36,445 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,445 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:36,445 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:36,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:36,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:36,445 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:36,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:36,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:36,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:36,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:36,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:36,448 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:36,448 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:36,448 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:36,448 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:36,448 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:36,450 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,450 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:36,450 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:36,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:36,450 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:36,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:36,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:36,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:36,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,452 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:36,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:36,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:36,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:36,455 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:36,455 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:36,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,455 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:36,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:36,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:36,455 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:36,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:36,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:36,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,455 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:36,556 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:36,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:36,558 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:36,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:36,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:36,559 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:36,559 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:36,559 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:36,559 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:36,560 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,560 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,561 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:36,561 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:36,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:36,561 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,562 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:36,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:36,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:36,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:36,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:36,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:36,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:36,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:36,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:36,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:36,563 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:36,563 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:36,563 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:36,563 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:36,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:36,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:36,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:36,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:36,565 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,565 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:36,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:36,565 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:36,565 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:36,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:36,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:36,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:36,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f665253[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:36,567 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:36,567 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,568 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,568 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:36,568 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:36,568 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:36,568 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,568 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:36,570 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,570 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:36,570 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:36,571 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:36,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:36,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:36,573 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:36,573 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:36,573 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:36,573 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:36,573 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:36,575 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,575 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:36,575 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:36,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:36,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:36,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:36,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:36,576 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:36,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,577 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:36,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:36,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:36,582 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:36,582 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:36,582 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:36,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:36,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:36,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,582 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:36,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,582 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:36,583 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:36,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:36,683 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:36,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:36,685 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:36,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:36,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:36,686 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:36,686 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:36,686 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:36,686 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:36,687 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,687 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,689 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,689 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:36,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:36,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:36,689 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,689 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:36,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:36,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:36,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:36,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:36,690 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:36,690 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:36,690 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:36,691 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:36,691 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:36,691 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:36,691 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:36,691 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:36,691 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:36,691 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:36,691 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:36,691 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,691 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:36,691 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:36,692 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,692 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:36,693 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,693 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:36,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:36,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24aa6e15[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:36,694 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:36,694 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,695 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:36,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:36,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:36,696 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,696 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:36,697 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,697 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:36,697 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:36,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:36,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:36,697 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:36,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:36,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:36,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:36,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:36,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:36,700 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:36,700 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:36,700 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:36,700 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:36,700 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:36,702 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,702 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:36,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:36,702 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:36,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:36,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:36,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:36,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,703 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:36,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,704 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:36,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,706 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,706 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:36,706 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:36,706 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:36,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:36,706 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:36,706 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:36,706 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:36,706 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,707 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:36,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:36,707 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,707 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:36,707 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,707 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:36,707 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:36,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 541 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042136724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,807 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:36,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:36,809 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:36,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:36,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:36,810 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:36,810 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:36,810 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:36,810 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:36,811 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,811 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,812 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:36,812 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:36,812 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,812 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:36,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:36,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:36,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:36,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:36,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:36,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:36,814 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:36,814 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:36,814 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:36,814 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:36,814 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:36,814 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:36,814 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:36,814 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:36,814 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,814 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:36,814 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:36,815 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,815 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:36,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:36,816 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,816 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:36,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:36,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:36,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:36,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:36,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70666fcc[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:36,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:36,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,820 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,820 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:36,820 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:36,820 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:36,820 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,820 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:36,821 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,821 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:36,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:36,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:36,821 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:36,821 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:36,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:36,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:36,824 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:36,824 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:36,824 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:36,824 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:36,824 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:36,825 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,825 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:36,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,825 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:36,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:36,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:36,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:36,826 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:36,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,827 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:36,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:36,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,829 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,829 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,829 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:36,830 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:36,829 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:36,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:36,830 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:36,830 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:36,830 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:36,830 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,830 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,830 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:36,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:36,830 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,830 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:36,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:36,830 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:36,848 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=745 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/6e177dd4d36b4ea2ad0b9a344ea2e76b 2023-04-21 01:54:36,852 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/681490a41f44431b87bcd14fefe3dee1 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/681490a41f44431b87bcd14fefe3dee1 2023-04-21 01:54:36,856 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/681490a41f44431b87bcd14fefe3dee1, entries=150, sequenceid=745, filesize=25.4 K 2023-04-21 01:54:36,857 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/c03a59cbce6a4bd48f6c976fba897e34 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/c03a59cbce6a4bd48f6c976fba897e34 2023-04-21 01:54:36,861 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/c03a59cbce6a4bd48f6c976fba897e34, entries=150, sequenceid=745, filesize=11.9 K 2023-04-21 01:54:36,862 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/6e177dd4d36b4ea2ad0b9a344ea2e76b as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/6e177dd4d36b4ea2ad0b9a344ea2e76b 2023-04-21 01:54:36,866 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/6e177dd4d36b4ea2ad0b9a344ea2e76b, entries=150, sequenceid=745, filesize=11.9 K 2023-04-21 01:54:36,866 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for e00828c94a25954f7bdc6de3ac3e2662 in 1267ms, sequenceid=745, compaction requested=true 2023-04-21 01:54:36,866 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:36,866 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:36,866 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:54:36,867 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:54:36,866 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:36,867 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:36,868 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104832 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:54:36,868 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:54:36,868 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): e00828c94a25954f7bdc6de3ac3e2662/A is initiating minor compaction (all files) 2023-04-21 01:54:36,868 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): e00828c94a25954f7bdc6de3ac3e2662/B is initiating minor compaction (all files) 2023-04-21 01:54:36,868 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e00828c94a25954f7bdc6de3ac3e2662/A in TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:36,868 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e00828c94a25954f7bdc6de3ac3e2662/B in TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:36,868 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/a136d8b7b88545a68d8c4e3b976772dd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/ac02532900614daba46fd4f65c055b8a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/23de29cfc19b49fd87bc68ee2a853fb1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/681490a41f44431b87bcd14fefe3dee1] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp, totalSize=102.4 K 2023-04-21 01:54:36,868 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/7e1ae28fdada4fb0b97f542bf0253c29, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/bed30adb0e9c4378a4fd63be8c5ac17e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/45d25955ed134c50a1d4fab00c2aaaa4, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/c03a59cbce6a4bd48f6c976fba897e34] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp, totalSize=48.2 K 2023-04-21 01:54:36,869 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting a136d8b7b88545a68d8c4e3b976772dd, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1682042073006 2023-04-21 01:54:36,869 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 7e1ae28fdada4fb0b97f542bf0253c29, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1682042073006 2023-04-21 01:54:36,869 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting ac02532900614daba46fd4f65c055b8a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=695, earliestPutTs=1682042073142 2023-04-21 01:54:36,869 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting bed30adb0e9c4378a4fd63be8c5ac17e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=695, earliestPutTs=1682042073142 2023-04-21 01:54:36,870 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 23de29cfc19b49fd87bc68ee2a853fb1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=708, earliestPutTs=1682042073791 2023-04-21 01:54:36,870 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 45d25955ed134c50a1d4fab00c2aaaa4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=708, earliestPutTs=1682042073791 2023-04-21 01:54:36,870 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 681490a41f44431b87bcd14fefe3dee1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=745, earliestPutTs=1682042074976 2023-04-21 01:54:36,870 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting c03a59cbce6a4bd48f6c976fba897e34, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=745, earliestPutTs=1682042074976 2023-04-21 01:54:36,879 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): e00828c94a25954f7bdc6de3ac3e2662#B#compaction#212 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:36,888 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): e00828c94a25954f7bdc6de3ac3e2662#A#compaction#211 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:36,930 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:36,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:36,932 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:36,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:36,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:36,933 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:36,933 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:36,933 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:36,933 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:36,934 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,934 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,936 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,936 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:36,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:36,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:36,936 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:36,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,936 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:36,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:36,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:36,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:36,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:36,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:36,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:36,937 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:36,937 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:36,937 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:36,937 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:36,937 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:36,937 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:36,938 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:36,938 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e00828c94a25954f7bdc6de3ac3e2662 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-21 01:54:36,938 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=A 2023-04-21 01:54:36,938 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:36,938 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=B 2023-04-21 01:54:36,938 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:36,938 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=C 2023-04-21 01:54:36,938 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:36,953 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304218d9d44db2eae4458b27de633bcdc9827 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304218d9d44db2eae4458b27de633bcdc9827 2023-04-21 01:54:36,960 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=754, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/285d70aa4a984a54b9ce3a17c3fd5360 2023-04-21 01:54:36,982 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=754 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/e2171fc2e7134455b11d4c4090f803cd 2023-04-21 01:54:36,992 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=754 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/dcb5af9ded8d4a10833c01abe9893b04 2023-04-21 01:54:36,996 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/285d70aa4a984a54b9ce3a17c3fd5360 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/285d70aa4a984a54b9ce3a17c3fd5360 2023-04-21 01:54:37,001 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/285d70aa4a984a54b9ce3a17c3fd5360, entries=150, sequenceid=754, filesize=25.4 K 2023-04-21 01:54:37,002 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/e2171fc2e7134455b11d4c4090f803cd as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/e2171fc2e7134455b11d4c4090f803cd 2023-04-21 01:54:37,005 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/e2171fc2e7134455b11d4c4090f803cd, entries=150, sequenceid=754, filesize=11.9 K 2023-04-21 01:54:37,006 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/dcb5af9ded8d4a10833c01abe9893b04 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/dcb5af9ded8d4a10833c01abe9893b04 2023-04-21 01:54:37,009 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/dcb5af9ded8d4a10833c01abe9893b04, entries=150, sequenceid=754, filesize=11.9 K 2023-04-21 01:54:37,010 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for e00828c94a25954f7bdc6de3ac3e2662 in 72ms, sequenceid=754, compaction requested=true 2023-04-21 01:54:37,010 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:37,010 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:37,010 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:37,010 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:37,010 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,010 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:37,011 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:37,012 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,012 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,013 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,013 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:37,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:37,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c3e703f[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:37,014 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:37,014 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,015 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,016 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:37,016 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:37,016 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:37,016 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,016 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:37,017 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,017 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:37,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,017 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:37,017 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:37,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:37,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:37,020 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:37,020 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:37,020 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:37,020 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:37,020 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:37,022 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,022 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,022 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:37,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:37,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:37,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,023 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,025 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,028 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,028 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,028 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,028 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,028 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:37,028 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:37,028 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:37,028 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,028 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,028 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:37,028 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:37,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:37,028 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:37,028 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:37,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:37,028 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:37,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:37,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:37,128 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:37,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:37,131 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:37,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:37,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:37,132 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:37,132 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:37,132 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:37,132 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:37,133 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,133 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,138 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:37,138 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:37,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:37,139 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,139 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:37,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:37,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:37,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:37,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:37,140 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:37,140 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:37,140 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:37,144 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:37,144 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:37,144 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:37,144 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:37,144 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:37,144 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:37,144 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:37,144 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:37,144 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,144 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:37,144 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:37,147 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,147 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,147 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,147 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:37,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:37,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1075721[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:37,149 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:37,149 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,150 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,150 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:37,150 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:37,150 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:37,151 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,151 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:37,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:37,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:37,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:37,152 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:37,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:37,154 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:37,154 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:37,154 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:37,154 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:37,154 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:37,156 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,156 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:37,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,156 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:37,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:37,157 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,158 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,161 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:37,161 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:37,161 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:37,161 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,161 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,161 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,161 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,161 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,161 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,161 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:37,161 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:37,162 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:37,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:37,162 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:37,162 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:37,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:37,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:37,262 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:37,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:37,265 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:37,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:37,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:37,266 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:37,266 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:37,266 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:37,267 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:37,267 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,267 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,268 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:37,268 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:37,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:37,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,269 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,269 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:37,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:37,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:37,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:37,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:37,270 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:37,270 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:37,270 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:37,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:37,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:37,271 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:37,271 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:37,271 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:37,271 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:37,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:37,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:37,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:37,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:37,273 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,273 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,273 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,273 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:37,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:37,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76aa4bd2[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:37,275 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:37,275 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,276 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,276 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:37,277 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:37,277 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:37,277 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,277 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:37,279 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,279 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:37,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,279 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:37,279 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:37,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:37,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:37,282 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:37,282 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:37,282 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:37,282 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:37,282 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:37,284 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,284 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,284 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:37,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:37,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:37,285 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,286 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,289 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:37,289 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:37,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:37,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:37,289 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,289 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,290 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,290 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,290 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,290 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:37,290 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:37,290 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:37,290 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:37,290 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:37,290 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,290 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:37,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:37,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:37,296 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/1473c63326e648a2b0b916979d8fdd6d as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/1473c63326e648a2b0b916979d8fdd6d 2023-04-21 01:54:37,302 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e00828c94a25954f7bdc6de3ac3e2662/B of e00828c94a25954f7bdc6de3ac3e2662 into 1473c63326e648a2b0b916979d8fdd6d(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:37,302 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:37,302 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., storeName=e00828c94a25954f7bdc6de3ac3e2662/B, priority=12, startTime=1682042076866; duration=0sec 2023-04-21 01:54:37,302 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:37,302 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-21 01:54:37,304 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61578 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-21 01:54:37,304 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): e00828c94a25954f7bdc6de3ac3e2662/C is initiating minor compaction (all files) 2023-04-21 01:54:37,304 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e00828c94a25954f7bdc6de3ac3e2662/C in TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:37,305 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/f2572b68c8844941a36120c5530c77bb, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/dd7dea90249f407b84f0c920ab0a6310, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/e16255e566eb41a49d9d609529caa83d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/6e177dd4d36b4ea2ad0b9a344ea2e76b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/dcb5af9ded8d4a10833c01abe9893b04] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp, totalSize=60.1 K 2023-04-21 01:54:37,305 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting f2572b68c8844941a36120c5530c77bb, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1682042073006 2023-04-21 01:54:37,305 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting dd7dea90249f407b84f0c920ab0a6310, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=695, earliestPutTs=1682042073142 2023-04-21 01:54:37,306 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting e16255e566eb41a49d9d609529caa83d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=708, earliestPutTs=1682042073791 2023-04-21 01:54:37,306 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 6e177dd4d36b4ea2ad0b9a344ea2e76b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=745, earliestPutTs=1682042074976 2023-04-21 01:54:37,307 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting dcb5af9ded8d4a10833c01abe9893b04, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=754, earliestPutTs=1682042075607 2023-04-21 01:54:37,318 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): e00828c94a25954f7bdc6de3ac3e2662#C#compaction#216 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:37,350 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/6c066df4554e4e8c90742d2f441e1008 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/6c066df4554e4e8c90742d2f441e1008 2023-04-21 01:54:37,357 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e00828c94a25954f7bdc6de3ac3e2662/C of e00828c94a25954f7bdc6de3ac3e2662 into 6c066df4554e4e8c90742d2f441e1008(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:37,357 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:37,357 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., storeName=e00828c94a25954f7bdc6de3ac3e2662/C, priority=11, startTime=1682042076867; duration=0sec 2023-04-21 01:54:37,357 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:37,390 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:37,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:37,393 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:37,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:37,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:37,394 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:37,394 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:37,394 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:37,394 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:37,395 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,395 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,396 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:37,396 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:37,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:37,397 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,397 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:37,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:37,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,398 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:37,421 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:37,421 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:37,421 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:37,421 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:37,421 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:37,424 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:37,424 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:37,424 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:37,425 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:37,425 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:37,426 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:37,426 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:37,426 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:37,426 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,426 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:37,426 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:37,428 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,428 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,429 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,429 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:37,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:37,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35f152c4[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:37,431 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:37,431 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,432 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,432 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:37,432 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:37,432 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:37,432 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,432 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:37,433 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:37,434 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:37,434 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:37,434 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:37,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:37,437 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:37,437 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:37,437 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:37,437 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:37,437 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:37,439 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,439 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:37,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:37,439 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,440 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,442 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,451 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,451 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,451 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:37,451 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:37,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:37,451 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:37,452 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:37,451 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:37,451 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,452 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:37,452 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:37,452 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:37,452 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,452 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,452 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,552 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:37,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:37,555 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:37,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:37,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:37,557 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:37,557 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:37,557 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:37,557 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:37,557 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,557 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,558 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,558 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:37,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:37,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:37,559 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,559 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:37,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:37,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,560 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:37,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:37,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:37,560 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:37,560 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:37,560 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:37,561 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:37,561 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:37,561 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:37,561 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:37,561 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:37,561 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:37,561 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:37,561 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:37,561 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,561 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:37,561 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:37,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,563 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:37,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a7163cd[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:37,566 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:37,566 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,571 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,571 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:37,571 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:37,571 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:37,572 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,572 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:37,573 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,573 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:37,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,573 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:37,573 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:37,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:37,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:37,578 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:37,578 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:37,578 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:37,578 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:37,578 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:37,579 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,579 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,579 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:37,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:37,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:37,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,580 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,582 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,584 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,584 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:37,584 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:37,584 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:37,584 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,585 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,585 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,585 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,585 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,585 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:37,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:37,584 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:37,585 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:37,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:37,585 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:37,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:37,585 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:37,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:37,685 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:37,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:37,688 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:37,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:37,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:37,689 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:37,689 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:37,690 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:37,690 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:37,690 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,690 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,692 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,692 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:37,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:37,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:37,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,693 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,693 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:37,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:37,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:37,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:37,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:37,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:37,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:37,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:37,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:37,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:37,695 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:37,695 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:37,695 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:37,695 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:37,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:37,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:37,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:37,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:37,696 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,697 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,697 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:37,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:37,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31f483fd[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:37,698 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:37,699 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,700 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,700 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:37,700 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:37,700 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,700 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:37,700 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:37,701 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:37,701 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:37,701 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:37,701 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:37,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:37,704 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:37,705 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:37,705 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:37,705 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:37,705 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:37,707 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,707 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:37,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,707 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,708 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,709 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,711 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,711 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:37,712 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:37,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:37,712 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:37,712 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:37,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:37,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:37,712 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:37,711 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,712 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,712 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,712 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:37,712 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,712 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:37,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:37,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:37,712 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,712 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,736 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/1aec85088cf947ed9a0ff113c664c7bb as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/1aec85088cf947ed9a0ff113c664c7bb 2023-04-21 01:54:37,740 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e00828c94a25954f7bdc6de3ac3e2662/A of e00828c94a25954f7bdc6de3ac3e2662 into 1aec85088cf947ed9a0ff113c664c7bb(size=26.2 K), total size for store is 51.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:37,740 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:37,740 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., storeName=e00828c94a25954f7bdc6de3ac3e2662/A, priority=12, startTime=1682042076866; duration=0sec 2023-04-21 01:54:37,740 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:37,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:37,802 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e00828c94a25954f7bdc6de3ac3e2662 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-21 01:54:37,802 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=A 2023-04-21 01:54:37,802 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:37,802 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=B 2023-04-21 01:54:37,802 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:37,802 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=C 2023-04-21 01:54:37,802 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:37,836 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:37,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:37,840 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:37,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:37,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:37,842 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:37,842 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:37,842 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:37,842 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:37,843 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,843 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,844 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,844 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:37,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:37,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:37,845 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,845 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:37,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:37,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:37,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:37,848 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:37,848 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:37,848 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:37,851 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:37,851 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:37,851 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:37,851 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:37,851 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:37,852 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:37,852 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:37,852 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:37,852 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,852 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:37,852 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:37,853 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,854 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,854 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,854 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:37,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,856 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421ce4de164655e4ff9a91c2e42ad4cd4f6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421ce4de164655e4ff9a91c2e42ad4cd4f6 2023-04-21 01:54:37,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:37,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1524aaca[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:37,857 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:37,857 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,859 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,859 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,859 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:37,860 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:37,860 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:37,860 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:37,861 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,861 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:37,861 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:37,861 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:37,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:37,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:37,864 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:37,864 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:37,864 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:37,864 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:37,864 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:37,865 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,865 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:37,865 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,866 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:37,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:37,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,867 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,869 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,869 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,869 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,869 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:37,869 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:37,869 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:37,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:37,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:37,869 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:37,870 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:37,869 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,870 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,870 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,870 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:37,870 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:37,870 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:37,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:37,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:37,873 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=772, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/38a94651b08d466ebad2227cf8b259a7 2023-04-21 01:54:37,886 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=772 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/4fb7c724b12c4ab1b6ef7019ba7213fd 2023-04-21 01:54:37,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042077880, max timestamp=1682042077890Num uniques 859; 2023-04-21 01:54:37,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:54:37,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042077880, max timestamp=1682042077890Num uniques 859; 2023-04-21 01:54:37,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042077880, max timestamp=1682042077890Num uniques 859; 2023-04-21 01:54:37,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:54:37,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:54:37,898 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=772 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/145d3076fe984c34bf3d249de1111c17 2023-04-21 01:54:37,903 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/38a94651b08d466ebad2227cf8b259a7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/38a94651b08d466ebad2227cf8b259a7 2023-04-21 01:54:37,906 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/38a94651b08d466ebad2227cf8b259a7, entries=250, sequenceid=772, filesize=39.2 K 2023-04-21 01:54:37,907 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/4fb7c724b12c4ab1b6ef7019ba7213fd as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/4fb7c724b12c4ab1b6ef7019ba7213fd 2023-04-21 01:54:37,912 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/4fb7c724b12c4ab1b6ef7019ba7213fd, entries=150, sequenceid=772, filesize=11.9 K 2023-04-21 01:54:37,912 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/145d3076fe984c34bf3d249de1111c17 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/145d3076fe984c34bf3d249de1111c17 2023-04-21 01:54:37,917 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/145d3076fe984c34bf3d249de1111c17, entries=150, sequenceid=772, filesize=11.9 K 2023-04-21 01:54:37,917 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=80.51 KB/82440 for e00828c94a25954f7bdc6de3ac3e2662 in 115ms, sequenceid=772, compaction requested=true 2023-04-21 01:54:37,917 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:37,917 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:37,917 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:37,917 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:37,917 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:37,917 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:37,918 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:37,918 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 93006 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:37,918 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): e00828c94a25954f7bdc6de3ac3e2662/B is initiating minor compaction (all files) 2023-04-21 01:54:37,919 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): e00828c94a25954f7bdc6de3ac3e2662/A is initiating minor compaction (all files) 2023-04-21 01:54:37,919 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e00828c94a25954f7bdc6de3ac3e2662/A in TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:37,919 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/1aec85088cf947ed9a0ff113c664c7bb, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/285d70aa4a984a54b9ce3a17c3fd5360, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/38a94651b08d466ebad2227cf8b259a7] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp, totalSize=90.8 K 2023-04-21 01:54:37,919 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e00828c94a25954f7bdc6de3ac3e2662/B in TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:37,919 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/1473c63326e648a2b0b916979d8fdd6d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/e2171fc2e7134455b11d4c4090f803cd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/4fb7c724b12c4ab1b6ef7019ba7213fd] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp, totalSize=36.5 K 2023-04-21 01:54:37,919 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 1aec85088cf947ed9a0ff113c664c7bb, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=745, earliestPutTs=1682042074976 2023-04-21 01:54:37,919 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 1473c63326e648a2b0b916979d8fdd6d, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=745, earliestPutTs=1682042074976 2023-04-21 01:54:37,920 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 285d70aa4a984a54b9ce3a17c3fd5360, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=754, earliestPutTs=1682042075607 2023-04-21 01:54:37,920 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 38a94651b08d466ebad2227cf8b259a7, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=772, earliestPutTs=1682042077772 2023-04-21 01:54:37,920 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting e2171fc2e7134455b11d4c4090f803cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=754, earliestPutTs=1682042075607 2023-04-21 01:54:37,921 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 4fb7c724b12c4ab1b6ef7019ba7213fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=772, earliestPutTs=1682042077772 2023-04-21 01:54:37,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:37,922 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e00828c94a25954f7bdc6de3ac3e2662 3/3 column families, dataSize=87.22 KB heapSize=245.53 KB 2023-04-21 01:54:37,922 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=A 2023-04-21 01:54:37,922 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:37,922 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=B 2023-04-21 01:54:37,922 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:37,922 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=C 2023-04-21 01:54:37,922 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:37,928 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): e00828c94a25954f7bdc6de3ac3e2662#B#compaction#221 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:37,946 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): e00828c94a25954f7bdc6de3ac3e2662#A#compaction#220 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:37,970 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:37,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:37,972 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:37,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:37,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:37,973 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:37,973 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:37,973 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:37,973 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:37,974 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,974 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,976 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,976 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:37,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:37,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:37,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:37,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 591 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042137975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,977 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,977 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:37,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:37,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,977 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:37,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:37,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:37,978 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:37,978 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:37,978 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:37,978 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:37,978 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:37,978 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:37,978 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:37,978 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:37,978 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:37,978 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:37,978 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:37,978 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,978 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:37,978 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:37,980 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,980 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,980 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,980 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:37,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:37,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2afcbdd8[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:37,982 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:37,982 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,983 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,983 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:37,983 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:37,983 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,983 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:37,983 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:37,989 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:37,989 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,989 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:37,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,989 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:37,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:37,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:37,991 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:37,991 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:37,991 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:37,991 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:37,991 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:37,995 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,995 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,995 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:37,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:37,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:37,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:37,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:37,996 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:37,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:37,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:37,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,997 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:37,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:37,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,999 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,999 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:37,999 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,999 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:37,999 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:37,999 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:37,999 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:37,999 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,999 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:37,999 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,000 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:37,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:37,999 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:37,999 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:38,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:38,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:38,000 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:38,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:38,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:38,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:38,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 593 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042138077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,100 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:38,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:38,102 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:38,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:38,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:38,103 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:38,103 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:38,103 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:38,103 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:38,104 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,104 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,105 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:38,105 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:38,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:38,106 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,106 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:38,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:38,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:38,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:38,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:38,107 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:38,107 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:38,107 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:38,107 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:38,107 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:38,107 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:38,107 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:38,108 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:38,108 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:38,108 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:38,108 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:38,108 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,108 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:38,108 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:38,111 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:38,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:38,111 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:38,112 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,112 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:38,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:38,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:38,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:38,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b12cc41[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:38,114 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:38,114 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,115 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,115 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:38,115 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:38,116 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:38,116 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,116 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:38,117 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:38,117 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:38,117 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:38,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:38,117 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:38,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:38,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:38,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:38,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:38,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:38,120 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:38,121 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:38,121 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:38,121 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:38,121 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:38,122 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,122 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,122 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:38,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:38,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:38,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:38,123 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:38,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:38,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:38,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,126 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:38,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,134 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:38,134 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,134 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,134 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:38,134 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:38,134 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:38,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:38,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:38,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:38,134 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,134 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:38,134 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,134 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:38,134 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,135 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:38,135 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,135 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:38,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:38,199 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:38,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57864 deadline: 1682042138198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,201 DEBUG [Thread-1014] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18272 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:54:38,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:38,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57852 deadline: 1682042138201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,205 DEBUG [Thread-1020] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18275 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:54:38,235 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:38,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:38,237 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:38,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:38,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:38,238 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:38,238 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:38,238 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:38,238 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:38,238 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,239 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,240 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:38,240 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:38,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:38,240 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,240 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:38,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:38,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:38,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:38,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:38,241 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:38,242 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:38,242 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:38,242 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:38,242 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:38,242 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:38,242 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:38,242 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:38,242 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:38,242 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:38,242 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:38,242 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,242 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:38,242 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:38,243 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:38,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:38,243 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:38,244 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,244 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:38,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:38,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:38,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:38,246 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:38,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e6c49b5[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:38,246 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,247 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,247 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:38,247 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:38,247 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,247 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:38,247 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:38,249 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,249 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:38,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:38,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:38,249 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:38,249 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:38,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:38,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:38,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:38,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:38,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:38,252 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:38,252 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:38,252 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:38,252 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:38,252 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:38,253 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,253 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:38,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:38,253 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:38,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:38,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:38,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:38,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,254 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:38,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,257 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:38,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,259 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:38,259 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:38,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:38,259 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:38,259 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:38,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:38,259 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,259 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:38,260 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,260 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:38,260 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:38,260 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:38,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:38,260 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,260 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,260 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,260 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:38,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 595 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042138280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,354 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421f85f79150c74405cb429c66921d893e1 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421f85f79150c74405cb429c66921d893e1 2023-04-21 01:54:38,360 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:38,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:38,361 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=802, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/dc8c8d6bac43452bbe2d4c5c30cf3b1e 2023-04-21 01:54:38,363 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:38,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:38,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:38,365 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:38,365 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:38,365 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:38,365 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:38,365 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,365 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,365 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/9c50525e83054ff588f70638c976ef16 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/9c50525e83054ff588f70638c976ef16 2023-04-21 01:54:38,366 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:38,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:38,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:38,366 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,367 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,367 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:38,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:38,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,368 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:38,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:38,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:38,368 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:38,368 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:38,368 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:38,369 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:38,369 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:38,369 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:38,369 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:38,369 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:38,369 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:38,369 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:38,369 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:38,369 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,369 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:38,369 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:38,372 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,372 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:38,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:38,372 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,372 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:38,372 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e00828c94a25954f7bdc6de3ac3e2662/B of e00828c94a25954f7bdc6de3ac3e2662 into 9c50525e83054ff588f70638c976ef16(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:38,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:38,372 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:38,372 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., storeName=e00828c94a25954f7bdc6de3ac3e2662/B, priority=13, startTime=1682042077917; duration=0sec 2023-04-21 01:54:38,373 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:38,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:38,373 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-21 01:54:38,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,374 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-21 01:54:38,374 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-21 01:54:38,374 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. because compaction request was cancelled 2023-04-21 01:54:38,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:38,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:38,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e2ed648[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:38,380 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:38,380 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,381 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,381 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:38,381 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:38,381 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:38,382 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,382 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:38,383 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,383 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:38,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:38,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:38,383 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:38,383 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:38,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:38,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:38,385 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=802 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/afa899697cc64d1a8e8037c24a20579c 2023-04-21 01:54:38,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:38,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:38,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:38,388 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:38,388 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:38,388 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:38,388 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:38,388 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:38,389 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,390 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:38,390 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:38,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:38,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:38,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:38,390 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:38,391 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:38,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,395 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:38,395 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:38,395 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:38,395 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,395 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:38,395 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:38,395 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:38,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:38,395 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,395 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,395 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,395 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,395 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:38,395 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,395 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:38,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:38,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:38,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:38,496 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:38,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:38,498 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:38,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:38,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:38,499 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:38,499 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:38,499 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:38,499 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:38,500 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,500 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,501 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,501 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:38,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:38,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:38,501 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,501 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:38,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:38,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:38,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:38,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:38,502 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:38,502 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:38,502 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:38,503 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:38,503 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:38,503 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:38,503 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:38,503 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:38,503 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:38,503 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:38,503 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:38,503 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,503 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:38,503 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:38,504 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,504 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:38,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:38,505 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,505 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:38,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:38,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:38,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:38,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:38,506 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:38,506 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4173a828[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:38,507 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,507 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:38,507 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:38,507 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,507 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:38,507 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:38,515 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:38,515 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,515 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:38,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:38,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:38,515 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:38,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:38,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:38,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:38,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:38,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:38,517 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:38,518 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:38,518 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:38,518 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:38,518 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:38,519 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,519 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,519 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:38,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:38,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:38,519 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:38,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:38,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:38,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:38,521 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:38,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,523 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,523 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,523 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,523 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:38,523 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:38,523 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:38,523 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:38,523 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:38,523 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,523 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:38,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:38,523 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,523 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,523 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:38,523 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:38,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:38,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:38,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:38,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:38,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 597 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57904 deadline: 1682042138582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,623 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:38,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:38,625 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:38,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:38,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:38,627 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:38,627 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:38,627 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:38,627 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:38,627 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,627 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,628 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:38,628 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:38,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:38,629 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,629 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:38,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:38,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:38,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:38,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:38,630 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:38,630 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:38,630 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:38,630 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:38,630 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:38,630 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. started... 2023-04-21 01:54:38,630 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:38,630 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. as already flushing 2023-04-21 01:54:38,630 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:38,630 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:38,630 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:38,630 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,630 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:38,630 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:38,632 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,632 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:38,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:38,633 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:38,633 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:38,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:38,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:38,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:38,634 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:38,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cbe0483[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:38,634 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,635 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,635 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:38,635 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:38,636 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:38,636 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,636 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:38,637 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,637 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:38,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:38,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:38,637 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:38,637 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:38,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:38,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:38,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:38,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:38,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:38,639 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:38,639 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:38,639 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:38,639 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:38,639 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:38,640 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,641 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,641 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:38,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:38,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:38,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:38,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:38,641 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:38,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:38,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,642 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:38,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:38,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,646 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,646 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,646 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:38,646 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:38,646 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:38,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:38,646 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:38,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:38,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:38,646 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:38,646 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:38,646 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:38,646 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,647 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:38,647 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:38,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:38,647 DEBUG [Thread-1024] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:38,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:38,647 DEBUG [Thread-1024] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:38,723 DEBUG [Thread-1027] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x421e5356 to 127.0.0.1:58383 2023-04-21 01:54:38,724 DEBUG [Thread-1027] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:54:38,725 DEBUG [Thread-1025] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x498694cb to 127.0.0.1:58383 2023-04-21 01:54:38,725 DEBUG [Thread-1025] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:54:38,727 DEBUG [Thread-1031] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a7b66a3 to 127.0.0.1:58383 2023-04-21 01:54:38,727 DEBUG [Thread-1031] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:54:38,728 DEBUG [Thread-1029] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4a0fd7b4 to 127.0.0.1:58383 2023-04-21 01:54:38,728 DEBUG [Thread-1029] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:54:38,747 DEBUG [Thread-1024] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:38,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:38,798 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=802 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/29295b4d9612486e8629f0edfa4b4318 2023-04-21 01:54:38,802 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/dc8c8d6bac43452bbe2d4c5c30cf3b1e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/dc8c8d6bac43452bbe2d4c5c30cf3b1e 2023-04-21 01:54:38,805 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/dc8c8d6bac43452bbe2d4c5c30cf3b1e, entries=250, sequenceid=802, filesize=39.2 K 2023-04-21 01:54:38,806 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/afa899697cc64d1a8e8037c24a20579c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/afa899697cc64d1a8e8037c24a20579c 2023-04-21 01:54:38,809 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/afa899697cc64d1a8e8037c24a20579c, entries=150, sequenceid=802, filesize=11.9 K 2023-04-21 01:54:38,810 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/29295b4d9612486e8629f0edfa4b4318 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/29295b4d9612486e8629f0edfa4b4318 2023-04-21 01:54:38,814 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/29295b4d9612486e8629f0edfa4b4318, entries=150, sequenceid=802, filesize=11.9 K 2023-04-21 01:54:38,814 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~245.48 KB/251376, currentSize=93.93 KB/96180 for e00828c94a25954f7bdc6de3ac3e2662 in 892ms, sequenceid=802, compaction requested=true 2023-04-21 01:54:38,815 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:38,815 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:38,815 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:38,815 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 3 compacting, 1 eligible, 16 blocking 2023-04-21 01:54:38,815 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-21 01:54:38,815 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-21 01:54:38,815 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-21 01:54:38,815 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. because compaction request was cancelled 2023-04-21 01:54:38,815 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:38,816 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:38,816 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): e00828c94a25954f7bdc6de3ac3e2662/C is initiating minor compaction (all files) 2023-04-21 01:54:38,816 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e00828c94a25954f7bdc6de3ac3e2662/C in TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:38,816 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/6c066df4554e4e8c90742d2f441e1008, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/145d3076fe984c34bf3d249de1111c17, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/29295b4d9612486e8629f0edfa4b4318] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp, totalSize=36.5 K 2023-04-21 01:54:38,817 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 6c066df4554e4e8c90742d2f441e1008, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=754, earliestPutTs=1682042075607 2023-04-21 01:54:38,817 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 145d3076fe984c34bf3d249de1111c17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=772, earliestPutTs=1682042077772 2023-04-21 01:54:38,817 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 29295b4d9612486e8629f0edfa4b4318, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=802, earliestPutTs=1682042077903 2023-04-21 01:54:38,822 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): e00828c94a25954f7bdc6de3ac3e2662#C#compaction#225 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:39,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:39,088 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e00828c94a25954f7bdc6de3ac3e2662 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-21 01:54:39,088 DEBUG [Thread-1018] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x24bd3872 to 127.0.0.1:58383 2023-04-21 01:54:39,088 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=A 2023-04-21 01:54:39,088 DEBUG [Thread-1018] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:54:39,088 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:39,088 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=B 2023-04-21 01:54:39,088 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:39,089 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=C 2023-04-21 01:54:39,089 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:39,182 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/c839893e63b94baa974a06e7b17df5fd as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/c839893e63b94baa974a06e7b17df5fd 2023-04-21 01:54:39,186 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e00828c94a25954f7bdc6de3ac3e2662/A of e00828c94a25954f7bdc6de3ac3e2662 into c839893e63b94baa974a06e7b17df5fd(size=26.3 K), total size for store is 65.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-21 01:54:39,186 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:39,186 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., storeName=e00828c94a25954f7bdc6de3ac3e2662/A, priority=13, startTime=1682042077917; duration=1sec 2023-04-21 01:54:39,186 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:39,186 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-21 01:54:39,186 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-21 01:54:39,187 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-21 01:54:39,187 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. because compaction request was cancelled 2023-04-21 01:54:39,234 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/6a2a293ad03b4b74a10342a343abd5c6 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/6a2a293ad03b4b74a10342a343abd5c6 2023-04-21 01:54:39,238 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e00828c94a25954f7bdc6de3ac3e2662/C of e00828c94a25954f7bdc6de3ac3e2662 into 6a2a293ad03b4b74a10342a343abd5c6(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:39,238 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:39,238 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., storeName=e00828c94a25954f7bdc6de3ac3e2662/C, priority=13, startTime=1682042078815; duration=0sec 2023-04-21 01:54:39,238 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:39,500 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421b3ba00a6addf4e6099103dba6eb84b20 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421b3ba00a6addf4e6099103dba6eb84b20 2023-04-21 01:54:39,905 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=821, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/647501f6440d4266a903bb732bd7a5e3 2023-04-21 01:54:40,313 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=821 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/d618171021ce48738910a01e5e83b11e 2023-04-21 01:54:40,722 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=821 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/592db6c3be024b37be0582663387cc5f 2023-04-21 01:54:40,728 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/647501f6440d4266a903bb732bd7a5e3 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/647501f6440d4266a903bb732bd7a5e3 2023-04-21 01:54:40,732 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/647501f6440d4266a903bb732bd7a5e3, entries=150, sequenceid=821, filesize=25.4 K 2023-04-21 01:54:40,733 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/d618171021ce48738910a01e5e83b11e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/d618171021ce48738910a01e5e83b11e 2023-04-21 01:54:40,737 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/d618171021ce48738910a01e5e83b11e, entries=150, sequenceid=821, filesize=11.9 K 2023-04-21 01:54:40,737 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/592db6c3be024b37be0582663387cc5f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/592db6c3be024b37be0582663387cc5f 2023-04-21 01:54:40,741 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/592db6c3be024b37be0582663387cc5f, entries=150, sequenceid=821, filesize=11.9 K 2023-04-21 01:54:40,742 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for e00828c94a25954f7bdc6de3ac3e2662 in 1654ms, sequenceid=821, compaction requested=true 2023-04-21 01:54:40,742 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:40,742 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:40,742 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:40,742 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:40,742 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:40,742 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:40,743 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 93108 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:40,743 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:40,743 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): e00828c94a25954f7bdc6de3ac3e2662/A is initiating minor compaction (all files) 2023-04-21 01:54:40,743 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): e00828c94a25954f7bdc6de3ac3e2662/B is initiating minor compaction (all files) 2023-04-21 01:54:40,743 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e00828c94a25954f7bdc6de3ac3e2662/A in TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:40,743 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e00828c94a25954f7bdc6de3ac3e2662/B in TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:40,743 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/c839893e63b94baa974a06e7b17df5fd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/dc8c8d6bac43452bbe2d4c5c30cf3b1e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/647501f6440d4266a903bb732bd7a5e3] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp, totalSize=90.9 K 2023-04-21 01:54:40,743 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/9c50525e83054ff588f70638c976ef16, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/afa899697cc64d1a8e8037c24a20579c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/d618171021ce48738910a01e5e83b11e] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp, totalSize=36.6 K 2023-04-21 01:54:40,744 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting c839893e63b94baa974a06e7b17df5fd, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=772, earliestPutTs=1682042077772 2023-04-21 01:54:40,744 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 9c50525e83054ff588f70638c976ef16, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=772, earliestPutTs=1682042077772 2023-04-21 01:54:40,744 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting dc8c8d6bac43452bbe2d4c5c30cf3b1e, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=802, earliestPutTs=1682042077899 2023-04-21 01:54:40,744 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting afa899697cc64d1a8e8037c24a20579c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=802, earliestPutTs=1682042077903 2023-04-21 01:54:40,744 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 647501f6440d4266a903bb732bd7a5e3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=821, earliestPutTs=1682042077960 2023-04-21 01:54:40,744 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting d618171021ce48738910a01e5e83b11e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=821, earliestPutTs=1682042077960 2023-04-21 01:54:40,751 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): e00828c94a25954f7bdc6de3ac3e2662#B#compaction#229 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:40,754 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): e00828c94a25954f7bdc6de3ac3e2662#A#compaction#230 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:40,756 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-21 01:54:40,759 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/949b7b7a7cd44020aeb56245cebdb8a4 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/949b7b7a7cd44020aeb56245cebdb8a4 2023-04-21 01:54:40,763 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e00828c94a25954f7bdc6de3ac3e2662/B of e00828c94a25954f7bdc6de3ac3e2662 into 949b7b7a7cd44020aeb56245cebdb8a4(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:40,763 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:40,763 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., storeName=e00828c94a25954f7bdc6de3ac3e2662/B, priority=13, startTime=1682042080742; duration=0sec 2023-04-21 01:54:40,763 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:40,763 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-21 01:54:40,764 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-21 01:54:40,764 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-21 01:54:40,764 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. because compaction request was cancelled 2023-04-21 01:54:41,973 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/cc28a3f9639446a488bc5f7758930df5 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/cc28a3f9639446a488bc5f7758930df5 2023-04-21 01:54:41,977 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e00828c94a25954f7bdc6de3ac3e2662/A of e00828c94a25954f7bdc6de3ac3e2662 into cc28a3f9639446a488bc5f7758930df5(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-21 01:54:41,977 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:41,977 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662., storeName=e00828c94a25954f7bdc6de3ac3e2662/A, priority=13, startTime=1682042080742; duration=1sec 2023-04-21 01:54:41,977 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:43,334 DEBUG [Thread-1022] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x210d1008 to 127.0.0.1:58383 2023-04-21 01:54:43,334 DEBUG [Thread-1022] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:54:43,402 DEBUG [Thread-1016] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02bab6b2 to 127.0.0.1:58383 2023-04-21 01:54:43,402 DEBUG [Thread-1016] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:54:48,268 DEBUG [Thread-1014] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34135972 to 127.0.0.1:58383 2023-04-21 01:54:48,268 DEBUG [Thread-1014] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:54:48,299 DEBUG [Thread-1020] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x370947cb to 127.0.0.1:58383 2023-04-21 01:54:48,300 DEBUG [Thread-1020] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:54:48,300 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-21 01:54:48,300 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 11 2023-04-21 01:54:48,300 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 117 2023-04-21 01:54:48,300 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 492 2023-04-21 01:54:48,300 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 12 2023-04-21 01:54:48,300 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 92 2023-04-21 01:54:48,300 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-21 01:54:48,300 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(398): read 4354 2023-04-21 01:54:48,300 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(398): read 4424 2023-04-21 01:54:48,300 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-21 01:54:48,300 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(402): scanned 1814 2023-04-21 01:54:48,300 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(403): verified 5442 rows 2023-04-21 01:54:48,300 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(402): scanned 1808 2023-04-21 01:54:48,300 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(403): verified 5409 rows 2023-04-21 01:54:48,300 INFO [Listener at localhost/39939] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-21 01:54:48,300 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52274650 to 127.0.0.1:58383 2023-04-21 01:54:48,301 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:54:48,302 INFO [Listener at localhost/39939] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-21 01:54:48,304 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-21 01:54:48,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-21 01:54:48,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-21 01:54:48,308 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042088308"}]},"ts":"1682042088308"} 2023-04-21 01:54:48,309 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-21 01:54:48,311 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-21 01:54:48,311 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e00828c94a25954f7bdc6de3ac3e2662, UNASSIGN}] 2023-04-21 01:54:48,313 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e00828c94a25954f7bdc6de3ac3e2662, UNASSIGN 2023-04-21 01:54:48,313 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=e00828c94a25954f7bdc6de3ac3e2662, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:48,313 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042088313"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682042088313"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682042088313"}]},"ts":"1682042088313"} 2023-04-21 01:54:48,314 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399}] 2023-04-21 01:54:48,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-21 01:54:48,466 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:48,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e00828c94a25954f7bdc6de3ac3e2662, disabling compactions & flushes 2023-04-21 01:54:48,466 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:48,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:48,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. after waiting 0 ms 2023-04-21 01:54:48,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:48,466 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing e00828c94a25954f7bdc6de3ac3e2662 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-21 01:54:48,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=A 2023-04-21 01:54:48,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:48,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=B 2023-04-21 01:54:48,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:48,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e00828c94a25954f7bdc6de3ac3e2662, store=C 2023-04-21 01:54:48,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:48,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-21 01:54:48,877 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042117877a31748c4a6285155429f3146b6b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042117877a31748c4a6285155429f3146b6b 2023-04-21 01:54:48,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-21 01:54:49,283 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=832, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/f23883a083e147f0818d91beebe43ef2 2023-04-21 01:54:49,291 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=832 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/4c4dcb2a6dd54252b4d920d30b15ff6b 2023-04-21 01:54:49,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-21 01:54:49,702 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=832 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/f95a5b0101144360b3d0e769f1c2683c 2023-04-21 01:54:49,705 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/A/f23883a083e147f0818d91beebe43ef2 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/f23883a083e147f0818d91beebe43ef2 2023-04-21 01:54:49,708 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/f23883a083e147f0818d91beebe43ef2, entries=100, sequenceid=832, filesize=18.6 K 2023-04-21 01:54:49,709 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/B/4c4dcb2a6dd54252b4d920d30b15ff6b as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/4c4dcb2a6dd54252b4d920d30b15ff6b 2023-04-21 01:54:49,712 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/4c4dcb2a6dd54252b4d920d30b15ff6b, entries=100, sequenceid=832, filesize=9.5 K 2023-04-21 01:54:49,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/.tmp/C/f95a5b0101144360b3d0e769f1c2683c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/f95a5b0101144360b3d0e769f1c2683c 2023-04-21 01:54:49,716 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/f95a5b0101144360b3d0e769f1c2683c, entries=100, sequenceid=832, filesize=9.5 K 2023-04-21 01:54:49,717 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for e00828c94a25954f7bdc6de3ac3e2662 in 1251ms, sequenceid=832, compaction requested=true 2023-04-21 01:54:49,737 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/fbc0f8c668444f9db87be37ec8ff75da, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/7e00c2cebf3d4c8189a76c392a6c484c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/4a55a36917f14340a7700ed09daa7987, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/30594e1a61754b6ea41c587a59d5d7ec, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/bed55f6840e64da59359cea49d668e5b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/2b5f7650ffa942d083f5b4ecef724614, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/06489ef226974324b8669898ec881ab0, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/044e2cb029584348b6b441caf0adf245, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/078cdea4549549a3aa27cc652009de60, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/5d2e66e890c74a54986b07e6db554e31, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/f04311d5aada4476946982fd20be5f93, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/c5c9a827457042db9ded9cb7e634c152, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/564a5beb388b4e49a2cda44823c00ae6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/41a3de37d1aa47759fba629083b008a8, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/84b9ef0e620640e68de465b83f6a907f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/c9b0cbf8c91c437186da477445afce09, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/b3e6b0b2ac434e3a8a116e2d1476fd40, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/b2770d97d2214656ad05fa01351a20a9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/6ec4139669b74983a03cd68e61acc74a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/a136d8b7b88545a68d8c4e3b976772dd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/ac02532900614daba46fd4f65c055b8a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/23de29cfc19b49fd87bc68ee2a853fb1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/1aec85088cf947ed9a0ff113c664c7bb, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/681490a41f44431b87bcd14fefe3dee1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/285d70aa4a984a54b9ce3a17c3fd5360, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/38a94651b08d466ebad2227cf8b259a7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/c839893e63b94baa974a06e7b17df5fd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/dc8c8d6bac43452bbe2d4c5c30cf3b1e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/647501f6440d4266a903bb732bd7a5e3] to archive 2023-04-21 01:54:49,738 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-21 01:54:49,739 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/fbc0f8c668444f9db87be37ec8ff75da to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/fbc0f8c668444f9db87be37ec8ff75da 2023-04-21 01:54:49,740 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/7e00c2cebf3d4c8189a76c392a6c484c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/7e00c2cebf3d4c8189a76c392a6c484c 2023-04-21 01:54:49,741 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/4a55a36917f14340a7700ed09daa7987 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/4a55a36917f14340a7700ed09daa7987 2023-04-21 01:54:49,742 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/30594e1a61754b6ea41c587a59d5d7ec to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/30594e1a61754b6ea41c587a59d5d7ec 2023-04-21 01:54:49,743 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/bed55f6840e64da59359cea49d668e5b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/bed55f6840e64da59359cea49d668e5b 2023-04-21 01:54:49,744 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/2b5f7650ffa942d083f5b4ecef724614 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/2b5f7650ffa942d083f5b4ecef724614 2023-04-21 01:54:49,745 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/06489ef226974324b8669898ec881ab0 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/06489ef226974324b8669898ec881ab0 2023-04-21 01:54:49,746 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/044e2cb029584348b6b441caf0adf245 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/044e2cb029584348b6b441caf0adf245 2023-04-21 01:54:49,746 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/078cdea4549549a3aa27cc652009de60 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/078cdea4549549a3aa27cc652009de60 2023-04-21 01:54:49,747 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/5d2e66e890c74a54986b07e6db554e31 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/5d2e66e890c74a54986b07e6db554e31 2023-04-21 01:54:49,748 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/f04311d5aada4476946982fd20be5f93 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/f04311d5aada4476946982fd20be5f93 2023-04-21 01:54:49,749 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/c5c9a827457042db9ded9cb7e634c152 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/c5c9a827457042db9ded9cb7e634c152 2023-04-21 01:54:49,750 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/564a5beb388b4e49a2cda44823c00ae6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/564a5beb388b4e49a2cda44823c00ae6 2023-04-21 01:54:49,751 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/41a3de37d1aa47759fba629083b008a8 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/41a3de37d1aa47759fba629083b008a8 2023-04-21 01:54:49,751 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/84b9ef0e620640e68de465b83f6a907f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/84b9ef0e620640e68de465b83f6a907f 2023-04-21 01:54:49,752 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/c9b0cbf8c91c437186da477445afce09 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/c9b0cbf8c91c437186da477445afce09 2023-04-21 01:54:49,753 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/b3e6b0b2ac434e3a8a116e2d1476fd40 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/b3e6b0b2ac434e3a8a116e2d1476fd40 2023-04-21 01:54:49,754 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/b2770d97d2214656ad05fa01351a20a9 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/b2770d97d2214656ad05fa01351a20a9 2023-04-21 01:54:49,755 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/6ec4139669b74983a03cd68e61acc74a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/6ec4139669b74983a03cd68e61acc74a 2023-04-21 01:54:49,756 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/a136d8b7b88545a68d8c4e3b976772dd to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/a136d8b7b88545a68d8c4e3b976772dd 2023-04-21 01:54:49,757 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/ac02532900614daba46fd4f65c055b8a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/ac02532900614daba46fd4f65c055b8a 2023-04-21 01:54:49,757 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/23de29cfc19b49fd87bc68ee2a853fb1 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/23de29cfc19b49fd87bc68ee2a853fb1 2023-04-21 01:54:49,758 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/1aec85088cf947ed9a0ff113c664c7bb to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/1aec85088cf947ed9a0ff113c664c7bb 2023-04-21 01:54:49,759 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/681490a41f44431b87bcd14fefe3dee1 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/681490a41f44431b87bcd14fefe3dee1 2023-04-21 01:54:49,760 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/285d70aa4a984a54b9ce3a17c3fd5360 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/285d70aa4a984a54b9ce3a17c3fd5360 2023-04-21 01:54:49,761 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/38a94651b08d466ebad2227cf8b259a7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/38a94651b08d466ebad2227cf8b259a7 2023-04-21 01:54:49,762 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/c839893e63b94baa974a06e7b17df5fd to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/c839893e63b94baa974a06e7b17df5fd 2023-04-21 01:54:49,763 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/dc8c8d6bac43452bbe2d4c5c30cf3b1e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/dc8c8d6bac43452bbe2d4c5c30cf3b1e 2023-04-21 01:54:49,763 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/647501f6440d4266a903bb732bd7a5e3 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/647501f6440d4266a903bb732bd7a5e3 2023-04-21 01:54:49,766 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/18a02e8bc0b344f9bca5a5febbfc65da, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/c13f074985214d19b03b3e475896cdbf, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/2f52a25ed41141a09da755973c49d355, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/d12924fab43644d1967d453255d5a488, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/39a50432a99647c2b1a74b8ffa73623f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/9d2b87195832418bb6c706bc49348be5, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/14268ab08d464ca49e7b2b021c9b8ec1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/1a5fca224cb049a683942fe74af11f77, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/4490a21120c543e8b713afffedf40fc3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/661aa6fbaf9946a28e570db8d82263c3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/194a62ec98de40a09d9b467abb214060, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/0ccd49879c3b4f44a069de4fbd971a96, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/a284ace671164f8dbfde7eafb3bf950a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/fc00e37fa6234c94bf0d82cf5e2bc213, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/feae6c9c5f8248938aa23dd5ae7804ec, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/88198a0899c34bd280e459a8adf73720, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/fa52ff9ca2194724b8047383b182f7ad, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/8da3d20fc464415e97607af9706d641d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/7e1ae28fdada4fb0b97f542bf0253c29, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/674ec3b4f40144ba89573c82c7327439, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/bed30adb0e9c4378a4fd63be8c5ac17e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/45d25955ed134c50a1d4fab00c2aaaa4, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/1473c63326e648a2b0b916979d8fdd6d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/c03a59cbce6a4bd48f6c976fba897e34, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/e2171fc2e7134455b11d4c4090f803cd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/9c50525e83054ff588f70638c976ef16, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/4fb7c724b12c4ab1b6ef7019ba7213fd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/afa899697cc64d1a8e8037c24a20579c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/d618171021ce48738910a01e5e83b11e] to archive 2023-04-21 01:54:49,767 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-21 01:54:49,768 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/18a02e8bc0b344f9bca5a5febbfc65da to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/18a02e8bc0b344f9bca5a5febbfc65da 2023-04-21 01:54:49,769 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/c13f074985214d19b03b3e475896cdbf to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/c13f074985214d19b03b3e475896cdbf 2023-04-21 01:54:49,769 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/2f52a25ed41141a09da755973c49d355 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/2f52a25ed41141a09da755973c49d355 2023-04-21 01:54:49,770 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/d12924fab43644d1967d453255d5a488 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/d12924fab43644d1967d453255d5a488 2023-04-21 01:54:49,771 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/39a50432a99647c2b1a74b8ffa73623f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/39a50432a99647c2b1a74b8ffa73623f 2023-04-21 01:54:49,772 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/9d2b87195832418bb6c706bc49348be5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/9d2b87195832418bb6c706bc49348be5 2023-04-21 01:54:49,772 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/14268ab08d464ca49e7b2b021c9b8ec1 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/14268ab08d464ca49e7b2b021c9b8ec1 2023-04-21 01:54:49,773 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/1a5fca224cb049a683942fe74af11f77 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/1a5fca224cb049a683942fe74af11f77 2023-04-21 01:54:49,774 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/4490a21120c543e8b713afffedf40fc3 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/4490a21120c543e8b713afffedf40fc3 2023-04-21 01:54:49,775 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/661aa6fbaf9946a28e570db8d82263c3 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/661aa6fbaf9946a28e570db8d82263c3 2023-04-21 01:54:49,776 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/194a62ec98de40a09d9b467abb214060 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/194a62ec98de40a09d9b467abb214060 2023-04-21 01:54:49,776 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/0ccd49879c3b4f44a069de4fbd971a96 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/0ccd49879c3b4f44a069de4fbd971a96 2023-04-21 01:54:49,777 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/a284ace671164f8dbfde7eafb3bf950a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/a284ace671164f8dbfde7eafb3bf950a 2023-04-21 01:54:49,778 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/fc00e37fa6234c94bf0d82cf5e2bc213 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/fc00e37fa6234c94bf0d82cf5e2bc213 2023-04-21 01:54:49,779 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/feae6c9c5f8248938aa23dd5ae7804ec to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/feae6c9c5f8248938aa23dd5ae7804ec 2023-04-21 01:54:49,780 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/88198a0899c34bd280e459a8adf73720 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/88198a0899c34bd280e459a8adf73720 2023-04-21 01:54:49,781 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/fa52ff9ca2194724b8047383b182f7ad to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/fa52ff9ca2194724b8047383b182f7ad 2023-04-21 01:54:49,782 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/8da3d20fc464415e97607af9706d641d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/8da3d20fc464415e97607af9706d641d 2023-04-21 01:54:49,783 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/7e1ae28fdada4fb0b97f542bf0253c29 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/7e1ae28fdada4fb0b97f542bf0253c29 2023-04-21 01:54:49,784 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/674ec3b4f40144ba89573c82c7327439 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/674ec3b4f40144ba89573c82c7327439 2023-04-21 01:54:49,785 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/bed30adb0e9c4378a4fd63be8c5ac17e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/bed30adb0e9c4378a4fd63be8c5ac17e 2023-04-21 01:54:49,786 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/45d25955ed134c50a1d4fab00c2aaaa4 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/45d25955ed134c50a1d4fab00c2aaaa4 2023-04-21 01:54:49,788 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/1473c63326e648a2b0b916979d8fdd6d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/1473c63326e648a2b0b916979d8fdd6d 2023-04-21 01:54:49,789 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/c03a59cbce6a4bd48f6c976fba897e34 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/c03a59cbce6a4bd48f6c976fba897e34 2023-04-21 01:54:49,790 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/e2171fc2e7134455b11d4c4090f803cd to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/e2171fc2e7134455b11d4c4090f803cd 2023-04-21 01:54:49,791 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/9c50525e83054ff588f70638c976ef16 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/9c50525e83054ff588f70638c976ef16 2023-04-21 01:54:49,792 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/4fb7c724b12c4ab1b6ef7019ba7213fd to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/4fb7c724b12c4ab1b6ef7019ba7213fd 2023-04-21 01:54:49,793 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/afa899697cc64d1a8e8037c24a20579c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/afa899697cc64d1a8e8037c24a20579c 2023-04-21 01:54:49,794 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/d618171021ce48738910a01e5e83b11e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/d618171021ce48738910a01e5e83b11e 2023-04-21 01:54:49,798 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/5091907d883d4b8fae3fd819660f3ef9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/8ae4f724d45b40e8bbca828c18e4a014, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/c6b32ede87ce4ee4bcf7ae7dcd7bf3bd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/00ab17fdc3994fec9f7a6ea69ac6f6cd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/8298aa1a5f0247b3b7b6e012b5b04219, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/296a14adbbff4ebbb7e25742b414e2ec, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/3adfb855ac3e4dd39653a2865359a96f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/98f4b7fcbb2245718e7454ab0747b10a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/35b27328c5034d72925cdd99902c0912, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/cf3fce13dfe348caaf9d67e444eb223e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/7aeb45bed1cf4677803e7db243749cf7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/454a6881f23e42e4b6148506c8e1c45c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/59c5fe3771794beb829e7d66b93175f7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/e1d2ab863c9547b7a4ed66c3c859bb20, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/7f8ada7d5bd748eaa5ca3675217885ec, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/8cd692f968a542938ab76574c308927c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/7e23253b73f54b54985dacbbfcdda605, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/5f22e79250734f55ab836d2c36ffd9fd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/f2572b68c8844941a36120c5530c77bb, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/31c7f8abbc564582b921d56c6a195c43, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/dd7dea90249f407b84f0c920ab0a6310, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/e16255e566eb41a49d9d609529caa83d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/6e177dd4d36b4ea2ad0b9a344ea2e76b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/6c066df4554e4e8c90742d2f441e1008, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/dcb5af9ded8d4a10833c01abe9893b04, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/145d3076fe984c34bf3d249de1111c17, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/29295b4d9612486e8629f0edfa4b4318] to archive 2023-04-21 01:54:49,798 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-21 01:54:49,800 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/5091907d883d4b8fae3fd819660f3ef9 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/5091907d883d4b8fae3fd819660f3ef9 2023-04-21 01:54:49,801 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/8ae4f724d45b40e8bbca828c18e4a014 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/8ae4f724d45b40e8bbca828c18e4a014 2023-04-21 01:54:49,803 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/c6b32ede87ce4ee4bcf7ae7dcd7bf3bd to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/c6b32ede87ce4ee4bcf7ae7dcd7bf3bd 2023-04-21 01:54:49,804 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/00ab17fdc3994fec9f7a6ea69ac6f6cd to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/00ab17fdc3994fec9f7a6ea69ac6f6cd 2023-04-21 01:54:49,805 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/8298aa1a5f0247b3b7b6e012b5b04219 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/8298aa1a5f0247b3b7b6e012b5b04219 2023-04-21 01:54:49,806 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/296a14adbbff4ebbb7e25742b414e2ec to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/296a14adbbff4ebbb7e25742b414e2ec 2023-04-21 01:54:49,807 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/3adfb855ac3e4dd39653a2865359a96f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/3adfb855ac3e4dd39653a2865359a96f 2023-04-21 01:54:49,809 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/98f4b7fcbb2245718e7454ab0747b10a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/98f4b7fcbb2245718e7454ab0747b10a 2023-04-21 01:54:49,810 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/35b27328c5034d72925cdd99902c0912 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/35b27328c5034d72925cdd99902c0912 2023-04-21 01:54:49,811 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/cf3fce13dfe348caaf9d67e444eb223e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/cf3fce13dfe348caaf9d67e444eb223e 2023-04-21 01:54:49,812 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/7aeb45bed1cf4677803e7db243749cf7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/7aeb45bed1cf4677803e7db243749cf7 2023-04-21 01:54:49,813 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/454a6881f23e42e4b6148506c8e1c45c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/454a6881f23e42e4b6148506c8e1c45c 2023-04-21 01:54:49,814 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/59c5fe3771794beb829e7d66b93175f7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/59c5fe3771794beb829e7d66b93175f7 2023-04-21 01:54:49,815 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/e1d2ab863c9547b7a4ed66c3c859bb20 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/e1d2ab863c9547b7a4ed66c3c859bb20 2023-04-21 01:54:49,816 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/7f8ada7d5bd748eaa5ca3675217885ec to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/7f8ada7d5bd748eaa5ca3675217885ec 2023-04-21 01:54:49,817 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/8cd692f968a542938ab76574c308927c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/8cd692f968a542938ab76574c308927c 2023-04-21 01:54:49,819 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/7e23253b73f54b54985dacbbfcdda605 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/7e23253b73f54b54985dacbbfcdda605 2023-04-21 01:54:49,820 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/5f22e79250734f55ab836d2c36ffd9fd to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/5f22e79250734f55ab836d2c36ffd9fd 2023-04-21 01:54:49,821 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/f2572b68c8844941a36120c5530c77bb to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/f2572b68c8844941a36120c5530c77bb 2023-04-21 01:54:49,822 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/31c7f8abbc564582b921d56c6a195c43 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/31c7f8abbc564582b921d56c6a195c43 2023-04-21 01:54:49,823 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/dd7dea90249f407b84f0c920ab0a6310 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/dd7dea90249f407b84f0c920ab0a6310 2023-04-21 01:54:49,824 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/e16255e566eb41a49d9d609529caa83d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/e16255e566eb41a49d9d609529caa83d 2023-04-21 01:54:49,825 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/6e177dd4d36b4ea2ad0b9a344ea2e76b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/6e177dd4d36b4ea2ad0b9a344ea2e76b 2023-04-21 01:54:49,826 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/6c066df4554e4e8c90742d2f441e1008 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/6c066df4554e4e8c90742d2f441e1008 2023-04-21 01:54:49,827 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/dcb5af9ded8d4a10833c01abe9893b04 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/dcb5af9ded8d4a10833c01abe9893b04 2023-04-21 01:54:49,828 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/145d3076fe984c34bf3d249de1111c17 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/145d3076fe984c34bf3d249de1111c17 2023-04-21 01:54:49,828 DEBUG [StoreCloser-TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/29295b4d9612486e8629f0edfa4b4318 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/29295b4d9612486e8629f0edfa4b4318 2023-04-21 01:54:49,832 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/recovered.edits/835.seqid, newMaxSeqId=835, maxSeqId=4 2023-04-21 01:54:49,834 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662. 2023-04-21 01:54:49,834 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e00828c94a25954f7bdc6de3ac3e2662: 2023-04-21 01:54:49,835 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:49,836 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=e00828c94a25954f7bdc6de3ac3e2662, regionState=CLOSED 2023-04-21 01:54:49,836 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042089836"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682042089836"}]},"ts":"1682042089836"} 2023-04-21 01:54:49,838 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-21 01:54:49,838 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure e00828c94a25954f7bdc6de3ac3e2662, server=jenkins-hbase4.apache.org,40131,1682042019399 in 1.5230 sec 2023-04-21 01:54:49,839 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-21 01:54:49,839 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e00828c94a25954f7bdc6de3ac3e2662, UNASSIGN in 1.5270 sec 2023-04-21 01:54:49,840 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042089840"}]},"ts":"1682042089840"} 2023-04-21 01:54:49,840 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-21 01:54:49,842 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-21 01:54:49,843 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5380 sec 2023-04-21 01:54:50,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-21 01:54:50,412 INFO [Listener at localhost/39939] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-21 01:54:50,413 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-21 01:54:50,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:54:50,415 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:54:50,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-21 01:54:50,416 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:54:50,420 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:50,422 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A, FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B, FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C, FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/recovered.edits] 2023-04-21 01:54:50,424 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/cc28a3f9639446a488bc5f7758930df5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/cc28a3f9639446a488bc5f7758930df5 2023-04-21 01:54:50,425 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/f23883a083e147f0818d91beebe43ef2 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/A/f23883a083e147f0818d91beebe43ef2 2023-04-21 01:54:50,427 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/4c4dcb2a6dd54252b4d920d30b15ff6b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/4c4dcb2a6dd54252b4d920d30b15ff6b 2023-04-21 01:54:50,429 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/949b7b7a7cd44020aeb56245cebdb8a4 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/B/949b7b7a7cd44020aeb56245cebdb8a4 2023-04-21 01:54:50,431 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/592db6c3be024b37be0582663387cc5f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/592db6c3be024b37be0582663387cc5f 2023-04-21 01:54:50,432 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/6a2a293ad03b4b74a10342a343abd5c6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/6a2a293ad03b4b74a10342a343abd5c6 2023-04-21 01:54:50,433 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/f95a5b0101144360b3d0e769f1c2683c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/C/f95a5b0101144360b3d0e769f1c2683c 2023-04-21 01:54:50,435 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/recovered.edits/835.seqid to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662/recovered.edits/835.seqid 2023-04-21 01:54:50,435 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/e00828c94a25954f7bdc6de3ac3e2662 2023-04-21 01:54:50,435 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-21 01:54:50,436 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-21 01:54:50,437 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-21 01:54:50,440 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421023616cbfc3d49eab843ef01fe6171f4 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421023616cbfc3d49eab843ef01fe6171f4 2023-04-21 01:54:50,441 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042107a7fe2e269a44069be058657c4cb3d1 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042107a7fe2e269a44069be058657c4cb3d1 2023-04-21 01:54:50,442 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042117877a31748c4a6285155429f3146b6b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042117877a31748c4a6285155429f3146b6b 2023-04-21 01:54:50,443 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042122d3ecc3ba4844889fa791faa261bdfc to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042122d3ecc3ba4844889fa791faa261bdfc 2023-04-21 01:54:50,444 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042124bf3829fd1d4191bfc8541a5fa8b8b6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042124bf3829fd1d4191bfc8541a5fa8b8b6 2023-04-21 01:54:50,445 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304212658eea2de5a442799e88b2fc15877d4 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304212658eea2de5a442799e88b2fc15877d4 2023-04-21 01:54:50,446 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304212bfc52f3bfdc46b980a554802969bc5e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304212bfc52f3bfdc46b980a554802969bc5e 2023-04-21 01:54:50,447 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304213f4bc8537518420f8eac24c2667ec2d4 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304213f4bc8537518420f8eac24c2667ec2d4 2023-04-21 01:54:50,447 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304214fac8fd2d2784be884e11ff0ebfe4c45 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304214fac8fd2d2784be884e11ff0ebfe4c45 2023-04-21 01:54:50,448 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304216dfce49ec0ae459d8687e2998e57f24c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304216dfce49ec0ae459d8687e2998e57f24c 2023-04-21 01:54:50,449 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042173a106fda05145ba8dc46c13145b46ca to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042173a106fda05145ba8dc46c13145b46ca 2023-04-21 01:54:50,450 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304218d9d44db2eae4458b27de633bcdc9827 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304218d9d44db2eae4458b27de633bcdc9827 2023-04-21 01:54:50,451 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304218e054822abc8434297030a3b7f57c07c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304218e054822abc8434297030a3b7f57c07c 2023-04-21 01:54:50,452 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421983d5b7892b54975b97dd1ef43dad697 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421983d5b7892b54975b97dd1ef43dad697 2023-04-21 01:54:50,453 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421986abb4c4f8a4779bdc0b3d3d5ba767e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421986abb4c4f8a4779bdc0b3d3d5ba767e 2023-04-21 01:54:50,454 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304219f792ba51b78407aa72fd5dca4edc76a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304219f792ba51b78407aa72fd5dca4edc76a 2023-04-21 01:54:50,455 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421b2cd07e5625247b6ba9a6a921b2ea1f6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421b2cd07e5625247b6ba9a6a921b2ea1f6 2023-04-21 01:54:50,456 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421b3ba00a6addf4e6099103dba6eb84b20 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421b3ba00a6addf4e6099103dba6eb84b20 2023-04-21 01:54:50,457 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421c426ccd42e074caba4001e87120f392e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421c426ccd42e074caba4001e87120f392e 2023-04-21 01:54:50,458 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421c44ccee88ec94ed2b11b48b841ebfaa6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421c44ccee88ec94ed2b11b48b841ebfaa6 2023-04-21 01:54:50,459 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421ce4de164655e4ff9a91c2e42ad4cd4f6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421ce4de164655e4ff9a91c2e42ad4cd4f6 2023-04-21 01:54:50,461 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421e1f97912952e44e4b761d2a041f14413 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421e1f97912952e44e4b761d2a041f14413 2023-04-21 01:54:50,462 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421f85f79150c74405cb429c66921d893e1 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421f85f79150c74405cb429c66921d893e1 2023-04-21 01:54:50,462 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-21 01:54:50,464 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:54:50,466 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-21 01:54:50,468 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-21 01:54:50,469 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:54:50,469 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-21 01:54:50,469 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682042090469"}]},"ts":"9223372036854775807"} 2023-04-21 01:54:50,471 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-21 01:54:50,471 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => e00828c94a25954f7bdc6de3ac3e2662, NAME => 'TestAcidGuarantees,,1682042055987.e00828c94a25954f7bdc6de3ac3e2662.', STARTKEY => '', ENDKEY => ''}] 2023-04-21 01:54:50,471 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-21 01:54:50,471 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682042090471"}]},"ts":"9223372036854775807"} 2023-04-21 01:54:50,472 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-21 01:54:50,474 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:54:50,475 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 61 msec 2023-04-21 01:54:50,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-21 01:54:50,516 INFO [Listener at localhost/39939] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-21 01:54:50,529 INFO [Listener at localhost/39939] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 229), OpenFileDescriptor=561 (was 560) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=484 (was 509), ProcessCount=169 (was 169), AvailableMemoryMB=8676 (was 8864) 2023-04-21 01:54:50,541 INFO [Listener at localhost/39939] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=561, MaxFileDescriptor=60000, SystemLoadAverage=484, ProcessCount=169, AvailableMemoryMB=8676 2023-04-21 01:54:50,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-21 01:54:50,543 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-21 01:54:50,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-21 01:54:50,545 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-21 01:54:50,546 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-21 01:54:50,546 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-21 01:54:50,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-21 01:54:50,547 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:54:50,547 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f empty. 2023-04-21 01:54:50,548 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:54:50,548 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-21 01:54:50,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-21 01:54:50,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-21 01:54:50,955 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-21 01:54:50,957 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 65bad43ef0e690f33e72a230ffd7ee4f, NAME => 'TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp 2023-04-21 01:54:51,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-21 01:54:51,363 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-21 01:54:51,363 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 65bad43ef0e690f33e72a230ffd7ee4f, disabling compactions & flushes 2023-04-21 01:54:51,363 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:51,363 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:51,363 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. after waiting 0 ms 2023-04-21 01:54:51,363 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:51,363 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:51,363 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:54:51,365 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-21 01:54:51,366 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042091366"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682042091366"}]},"ts":"1682042091366"} 2023-04-21 01:54:51,367 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-21 01:54:51,368 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-21 01:54:51,368 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042091368"}]},"ts":"1682042091368"} 2023-04-21 01:54:51,369 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-21 01:54:51,375 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=65bad43ef0e690f33e72a230ffd7ee4f, ASSIGN}] 2023-04-21 01:54:51,377 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=65bad43ef0e690f33e72a230ffd7ee4f, ASSIGN 2023-04-21 01:54:51,377 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=65bad43ef0e690f33e72a230ffd7ee4f, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40131,1682042019399; forceNewPlan=false, retain=false 2023-04-21 01:54:51,528 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=65bad43ef0e690f33e72a230ffd7ee4f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:51,529 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042091528"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682042091528"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682042091528"}]},"ts":"1682042091528"} 2023-04-21 01:54:51,530 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399}] 2023-04-21 01:54:51,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-21 01:54:51,685 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:51,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 65bad43ef0e690f33e72a230ffd7ee4f, NAME => 'TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.', STARTKEY => '', ENDKEY => ''} 2023-04-21 01:54:51,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:54:51,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-21 01:54:51,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:54:51,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:54:51,686 INFO [StoreOpener-65bad43ef0e690f33e72a230ffd7ee4f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:54:51,688 DEBUG [StoreOpener-65bad43ef0e690f33e72a230ffd7ee4f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A 2023-04-21 01:54:51,688 DEBUG [StoreOpener-65bad43ef0e690f33e72a230ffd7ee4f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A 2023-04-21 01:54:51,688 INFO [StoreOpener-65bad43ef0e690f33e72a230ffd7ee4f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:54:51,688 INFO [StoreOpener-65bad43ef0e690f33e72a230ffd7ee4f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 65bad43ef0e690f33e72a230ffd7ee4f columnFamilyName A 2023-04-21 01:54:51,689 INFO [StoreOpener-65bad43ef0e690f33e72a230ffd7ee4f-1] regionserver.HStore(310): Store=65bad43ef0e690f33e72a230ffd7ee4f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:54:51,689 INFO [StoreOpener-65bad43ef0e690f33e72a230ffd7ee4f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:54:51,690 DEBUG [StoreOpener-65bad43ef0e690f33e72a230ffd7ee4f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B 2023-04-21 01:54:51,690 DEBUG [StoreOpener-65bad43ef0e690f33e72a230ffd7ee4f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B 2023-04-21 01:54:51,690 INFO [StoreOpener-65bad43ef0e690f33e72a230ffd7ee4f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:54:51,691 INFO [StoreOpener-65bad43ef0e690f33e72a230ffd7ee4f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 65bad43ef0e690f33e72a230ffd7ee4f columnFamilyName B 2023-04-21 01:54:51,691 INFO [StoreOpener-65bad43ef0e690f33e72a230ffd7ee4f-1] regionserver.HStore(310): Store=65bad43ef0e690f33e72a230ffd7ee4f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:54:51,691 INFO [StoreOpener-65bad43ef0e690f33e72a230ffd7ee4f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:54:51,692 DEBUG [StoreOpener-65bad43ef0e690f33e72a230ffd7ee4f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C 2023-04-21 01:54:51,692 DEBUG [StoreOpener-65bad43ef0e690f33e72a230ffd7ee4f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C 2023-04-21 01:54:51,692 INFO [StoreOpener-65bad43ef0e690f33e72a230ffd7ee4f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:54:51,693 INFO [StoreOpener-65bad43ef0e690f33e72a230ffd7ee4f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 65bad43ef0e690f33e72a230ffd7ee4f columnFamilyName C 2023-04-21 01:54:51,693 INFO [StoreOpener-65bad43ef0e690f33e72a230ffd7ee4f-1] regionserver.HStore(310): Store=65bad43ef0e690f33e72a230ffd7ee4f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:54:51,693 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:51,694 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:54:51,694 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:54:51,696 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-21 01:54:51,696 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:54:51,698 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-21 01:54:51,698 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 65bad43ef0e690f33e72a230ffd7ee4f; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64047576, jitterRate=-0.045616745948791504}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-21 01:54:51,698 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:54:51,699 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., pid=30, masterSystemTime=1682042091681 2023-04-21 01:54:51,700 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:51,700 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:51,701 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=65bad43ef0e690f33e72a230ffd7ee4f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:51,701 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042091701"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682042091701"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682042091701"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682042091701"}]},"ts":"1682042091701"} 2023-04-21 01:54:51,703 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-21 01:54:51,703 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 in 172 msec 2023-04-21 01:54:51,704 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-21 01:54:51,704 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=65bad43ef0e690f33e72a230ffd7ee4f, ASSIGN in 328 msec 2023-04-21 01:54:51,705 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-21 01:54:51,705 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042091705"}]},"ts":"1682042091705"} 2023-04-21 01:54:51,706 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-21 01:54:51,708 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-21 01:54:51,709 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1650 sec 2023-04-21 01:54:52,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-21 01:54:52,651 INFO [Listener at localhost/39939] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-21 01:54:52,652 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x12e82edd to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:54:52,657 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ed1dde9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:54:52,658 DEBUG [Listener at localhost/39939] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:54:52,659 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52708, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:54:52,660 DEBUG [Listener at localhost/39939] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-21 01:54:52,662 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57342, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-21 01:54:52,663 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x23c97092 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:54:52,668 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70043cf0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:54:52,668 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x5c124820 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:54:52,671 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34b7fed0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:54:52,672 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x3157d4df to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:54:52,675 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f4ad99d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:54:52,675 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x3cf5bdb7 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:54:52,678 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@330b77da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:54:52,678 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x6919b2a0 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:54:52,683 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7810a231, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:54:52,684 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x08cd7cb0 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:54:52,689 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@753156b2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:54:52,689 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x0ea1ca03 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:54:52,694 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6dea0f9b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:54:52,694 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x433f251d to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:54:52,698 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@434e0eab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:54:52,698 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x66ee0bd0 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:54:52,704 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c43dddd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:54:52,704 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x3b4658f8 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:54:52,707 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2de064cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:54:52,712 DEBUG [hconnection-0x39150938-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:54:52,714 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:52,717 DEBUG [hconnection-0x73cba15a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:54:52,717 DEBUG [hconnection-0xe1f05fc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:54:52,717 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52724, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:54:52,717 DEBUG [hconnection-0x2a3cab38-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:54:52,718 DEBUG [hconnection-0x3e1fb1c7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:54:52,717 DEBUG [hconnection-0x316c5308-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:54:52,718 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52738, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:54:52,718 DEBUG [hconnection-0x74e7acb3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:54:52,718 DEBUG [hconnection-0x653cc715-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:54:52,719 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52772, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:54:52,720 DEBUG [hconnection-0x16273c96-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:54:52,721 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52802, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:54:52,726 DEBUG [hconnection-0x28acffe7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:54:52,732 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52732, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:54:52,732 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52752, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:54:52,732 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52762, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:54:52,732 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52782, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:54:52,732 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52788, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:54:52,732 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52812, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:54:52,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:52,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:52,735 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:52,735 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:52,736 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:52,736 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:52,736 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:52,736 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:52,737 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:52,737 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:52,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:52,737 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:52,738 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:52,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:52,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:52,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:52,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:52,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:52,738 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:52,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:52,738 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:52,738 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:52,738 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:52,739 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:52,739 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:52,739 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:52,739 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:54:52,742 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-21 01:54:52,744 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:54:52,744 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:52,744 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:54:52,744 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:52,744 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:54:52,744 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:52,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:54:52,746 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:54:52,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042092757, max timestamp=1682042092761Num uniques 859; 2023-04-21 01:54:52,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:54:52,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042092757, max timestamp=1682042092761Num uniques 859; 2023-04-21 01:54:52,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042092757, max timestamp=1682042092761Num uniques 859; 2023-04-21 01:54:52,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:54:52,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:54:52,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682042092769, max timestamp=1682042092781Num uniques 1018; 2023-04-21 01:54:52,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682042092769, max timestamp=1682042092781Num uniques 1018; 2023-04-21 01:54:52,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-21 01:54:52,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-21 01:54:52,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682042092776, max timestamp=1682042092782Num uniques 1018; 2023-04-21 01:54:52,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-21 01:54:52,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:52,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:52,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042152800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:52,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52724 deadline: 1682042152801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:52,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:52,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042152803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:52,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:52,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52802 deadline: 1682042152803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:52,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:52,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042152805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:52,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:52,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042152906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:52,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:52,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52724 deadline: 1682042152906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:52,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:52,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:52,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042152908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:52,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52802 deadline: 1682042152908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:52,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:52,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042152909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:53,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:53,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52724 deadline: 1682042153109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:53,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:53,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042153109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:53,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:53,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042153111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:53,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:53,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:53,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52802 deadline: 1682042153112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:53,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042153112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:53,188 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/c0067f0b089e4a189c6241100fd87dee 2023-04-21 01:54:53,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:53,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042153413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:53,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:53,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042153413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:53,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:53,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52724 deadline: 1682042153414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:53,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:53,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042153416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:53,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:53,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52802 deadline: 1682042153416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:53,600 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/ca3d6f55b9374776b92127793f0275cd 2023-04-21 01:54:53,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:53,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042153916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:53,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:53,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042153918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:53,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:53,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042153919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:53,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:53,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52724 deadline: 1682042153920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:53,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:53,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52802 deadline: 1682042153922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:54,010 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/04933a20731f4bd29c1c513b40bf8e07 2023-04-21 01:54:54,015 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/c0067f0b089e4a189c6241100fd87dee as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/c0067f0b089e4a189c6241100fd87dee 2023-04-21 01:54:54,018 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/c0067f0b089e4a189c6241100fd87dee, entries=150, sequenceid=10, filesize=11.6 K 2023-04-21 01:54:54,019 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/ca3d6f55b9374776b92127793f0275cd as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/ca3d6f55b9374776b92127793f0275cd 2023-04-21 01:54:54,023 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/ca3d6f55b9374776b92127793f0275cd, entries=150, sequenceid=10, filesize=11.6 K 2023-04-21 01:54:54,023 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/04933a20731f4bd29c1c513b40bf8e07 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/04933a20731f4bd29c1c513b40bf8e07 2023-04-21 01:54:54,026 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/04933a20731f4bd29c1c513b40bf8e07, entries=150, sequenceid=10, filesize=11.6 K 2023-04-21 01:54:54,027 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=143.13 KB/146560 for 65bad43ef0e690f33e72a230ffd7ee4f in 1287ms, sequenceid=10, compaction requested=false 2023-04-21 01:54:54,027 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:54:54,027 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:54,027 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:54,027 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:54,027 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:54,027 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:54,027 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:54,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:54,029 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:54,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:54,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:54,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:54,029 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:54,029 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:54,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:54,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:54,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2043cb40[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:54,031 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:54,031 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:54,033 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:54,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:54,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:54,033 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:54,033 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:54,034 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:54,033 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:54,034 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:54,035 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:54,035 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:54,035 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:54,035 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:54,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:54,037 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:54,037 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:54,037 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:54,038 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:54,038 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:54,039 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:54,039 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:54,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:54,039 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:54,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:54,040 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:54,041 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:54,043 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:54,044 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:54,044 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:54,044 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:54,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:54,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:54,044 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:54,044 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:54,044 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:54,044 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:54,044 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:54,044 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:54,044 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:54,044 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:54,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:54,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:54,044 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:54,044 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:54,144 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:54,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:54,147 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:54,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:54,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:54,148 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:54,148 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:54,148 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:54,148 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:54,149 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:54,149 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:54,150 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:54,150 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:54,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:54,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:54,151 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:54,151 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:54,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:54,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:54,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:54,151 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:54,151 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:54,151 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:54,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:54,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:54,152 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:54,152 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:54:54,152 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=143.13 KB heapSize=404.45 KB 2023-04-21 01:54:54,152 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:54:54,152 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:54,152 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:54:54,152 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:54,152 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:54:54,152 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:54,560 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/b6e9b995e479405b82455e914f7c0ead 2023-04-21 01:54:54,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:54:54,924 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:54:54,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:54,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:54,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042154927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:54,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52802 deadline: 1682042154927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:54,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:54,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042154928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:54,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:54,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042154929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:54,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:54,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52724 deadline: 1682042154931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:54,971 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/c2211c36751c4aa59fa22ee75bb280e5 2023-04-21 01:54:55,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:55,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042155032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:55,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042155032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:55,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042155032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:55,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52724 deadline: 1682042155034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:55,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042155235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:55,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042155235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:55,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042155236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:55,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52724 deadline: 1682042155238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,366 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-21 01:54:55,383 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/310a54e48c91461d8bac491a02a7a56a 2023-04-21 01:54:55,388 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/b6e9b995e479405b82455e914f7c0ead as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/b6e9b995e479405b82455e914f7c0ead 2023-04-21 01:54:55,392 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/b6e9b995e479405b82455e914f7c0ead, entries=150, sequenceid=64, filesize=11.6 K 2023-04-21 01:54:55,393 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/c2211c36751c4aa59fa22ee75bb280e5 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/c2211c36751c4aa59fa22ee75bb280e5 2023-04-21 01:54:55,396 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/c2211c36751c4aa59fa22ee75bb280e5, entries=150, sequenceid=64, filesize=11.6 K 2023-04-21 01:54:55,397 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/310a54e48c91461d8bac491a02a7a56a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/310a54e48c91461d8bac491a02a7a56a 2023-04-21 01:54:55,400 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/310a54e48c91461d8bac491a02a7a56a, entries=150, sequenceid=64, filesize=11.6 K 2023-04-21 01:54:55,401 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~143.13 KB/146560, heapSize ~404.41 KB/414112, currentSize=40.25 KB/41220 for 65bad43ef0e690f33e72a230ffd7ee4f in 1249ms, sequenceid=64, compaction requested=false 2023-04-21 01:54:55,401 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:54:55,401 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:55,401 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:55,401 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:55,401 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,402 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:55,402 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:55,403 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:55,403 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:55,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:55,403 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:55,403 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:55,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:55,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:55,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5827d596[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:55,405 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:55,405 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:55,406 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:55,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:55,406 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:55,406 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:55,406 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:55,406 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,406 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:55,413 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:55,413 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:55,413 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,413 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:55,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:55,416 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:55,416 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:55,416 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:55,416 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:55,416 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:55,417 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:55,417 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:55,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:55,417 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:55,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:55,418 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,419 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,421 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,421 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:55,421 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:55,421 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:55,421 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:55,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:55,422 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,422 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:55,422 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:55,422 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:55,422 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:55,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:55,422 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:55,422 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:55,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:55,422 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:55,422 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:55,522 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:55,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:55,524 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:55,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:55,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:55,527 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:55,527 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:55,527 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:55,527 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:55,527 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:55,527 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:55,528 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,528 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:55,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:55,529 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,529 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:55,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:55,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:55,530 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:55,530 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:55,530 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:55,530 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:55,530 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:55,530 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:55,530 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:55,530 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:55,531 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:54:55,531 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-21 01:54:55,531 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:54:55,531 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:55,531 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:54:55,531 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:55,531 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:54:55,531 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:55,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:54:55,540 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:54:55,542 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=73 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/3dd5c55c600849889a96441959c441d2 2023-04-21 01:54:55,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042095551, max timestamp=1682042095556Num uniques 859; 2023-04-21 01:54:55,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682042095551, max timestamp=1682042095556Num uniques 859; 2023-04-21 01:54:55,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:54:55,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682042095551, max timestamp=1682042095556Num uniques 859; 2023-04-21 01:54:55,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:54:55,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:54:55,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:55,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:55,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042155572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042155573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682042095567, max timestamp=1682042095575Num uniques 1068; 2023-04-21 01:54:55,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682042095567, max timestamp=1682042095575Num uniques 1068; 2023-04-21 01:54:55,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682042095567, max timestamp=1682042095575Num uniques 1018; 2023-04-21 01:54:55,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-21 01:54:55,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-21 01:54:55,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-21 01:54:55,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:55,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042155605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:55,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52724 deadline: 1682042155608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:55,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042155677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,681 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:55,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042155679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:55,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042155709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:55,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52724 deadline: 1682042155712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:55,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042155881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:55,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042155883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:55,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042155913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:55,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52724 deadline: 1682042155916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:55,960 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=73 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/9568a1a25dc9402ea6ed1503f59edcfa 2023-04-21 01:54:56,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:56,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042156184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:56,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042156184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:56,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042156218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:56,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52724 deadline: 1682042156220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,382 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=73 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/ef608552bce644c78fd173df6a833564 2023-04-21 01:54:56,387 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/3dd5c55c600849889a96441959c441d2 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/3dd5c55c600849889a96441959c441d2 2023-04-21 01:54:56,390 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/3dd5c55c600849889a96441959c441d2, entries=150, sequenceid=73, filesize=11.6 K 2023-04-21 01:54:56,391 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/9568a1a25dc9402ea6ed1503f59edcfa as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/9568a1a25dc9402ea6ed1503f59edcfa 2023-04-21 01:54:56,397 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/9568a1a25dc9402ea6ed1503f59edcfa, entries=150, sequenceid=73, filesize=11.6 K 2023-04-21 01:54:56,398 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/ef608552bce644c78fd173df6a833564 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/ef608552bce644c78fd173df6a833564 2023-04-21 01:54:56,402 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/ef608552bce644c78fd173df6a833564, entries=150, sequenceid=73, filesize=11.6 K 2023-04-21 01:54:56,403 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 65bad43ef0e690f33e72a230ffd7ee4f in 872ms, sequenceid=73, compaction requested=true 2023-04-21 01:54:56,403 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:54:56,403 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:56,403 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:56,403 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:56,403 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,403 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:56,403 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:56,406 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,406 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:56,406 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:56,406 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:56,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:56,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4490452[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:56,408 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:56,408 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:56,409 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:56,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:56,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:56,409 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:56,409 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,410 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:56,409 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:56,410 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:56,411 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:56,411 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,411 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:56,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:56,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:56,411 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:56,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:56,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:56,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:56,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:56,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:56,413 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:56,413 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:56,413 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:56,413 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:56,414 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:56,415 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:56,415 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:56,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:56,415 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:56,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:56,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:56,416 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:56,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:56,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:56,417 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:56,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,425 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,425 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:56,425 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:56,425 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,425 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:56,425 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:56,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:56,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:56,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:56,425 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:56,425 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:56,426 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:56,425 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:56,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:56,426 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:56,426 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:56,426 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:56,426 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:56,526 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:56,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:56,528 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:56,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:56,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:56,529 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:56,529 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:56,530 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:56,530 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:56,530 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:56,530 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:56,531 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:56,531 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:56,531 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,532 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:56,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:56,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:56,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:56,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:56,533 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:56,533 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:56,533 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:56,533 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:56,533 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:56,533 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:56,533 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:54:56,533 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=140.89 KB heapSize=399.38 KB 2023-04-21 01:54:56,533 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:54:56,533 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:56,533 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:54:56,534 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:56,534 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:54:56,534 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:56,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:54:56,689 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:54:56,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:56,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042156696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:56,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042156698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:56,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042156724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:56,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52724 deadline: 1682042156725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:56,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042156799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:56,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042156801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,942 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/014047aa59ec4fdba73ef1bcace58ea9 2023-04-21 01:54:56,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:56,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52802 deadline: 1682042156943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:56,946 DEBUG [Thread-1862] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:54:57,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:57,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042157002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:57,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042157003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:57,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042157307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:57,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042157308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,357 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/7cd5c498969c4695b9210c5d3de804d1 2023-04-21 01:54:57,367 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/8ac454416e604d27bef5d30cf8296e03 2023-04-21 01:54:57,371 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/014047aa59ec4fdba73ef1bcace58ea9 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/014047aa59ec4fdba73ef1bcace58ea9 2023-04-21 01:54:57,375 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/014047aa59ec4fdba73ef1bcace58ea9, entries=150, sequenceid=127, filesize=11.6 K 2023-04-21 01:54:57,375 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/7cd5c498969c4695b9210c5d3de804d1 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/7cd5c498969c4695b9210c5d3de804d1 2023-04-21 01:54:57,379 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/7cd5c498969c4695b9210c5d3de804d1, entries=150, sequenceid=127, filesize=11.6 K 2023-04-21 01:54:57,380 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/8ac454416e604d27bef5d30cf8296e03 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/8ac454416e604d27bef5d30cf8296e03 2023-04-21 01:54:57,383 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/8ac454416e604d27bef5d30cf8296e03, entries=150, sequenceid=127, filesize=11.6 K 2023-04-21 01:54:57,384 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~399.33 KB/408912, currentSize=40.25 KB/41220 for 65bad43ef0e690f33e72a230ffd7ee4f in 851ms, sequenceid=127, compaction requested=true 2023-04-21 01:54:57,384 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:54:57,384 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:57,384 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:57,384 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:57,384 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,384 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:57,384 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:57,385 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:57,385 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:57,386 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,386 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:57,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:57,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:57,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:57,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:57,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@85f8244[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:57,387 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:57,387 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,388 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,389 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:57,389 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:57,389 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:57,389 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,389 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:57,393 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,394 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:57,394 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:57,394 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:57,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:57,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:57,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:57,396 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:57,396 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:57,396 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:57,396 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:57,396 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:57,398 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:57,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:57,398 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,398 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:57,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:57,398 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:57,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:57,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:57,399 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:57,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,402 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:57,402 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:57,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:57,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:57,402 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,402 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:57,402 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:57,402 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,402 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,402 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,402 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,402 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:57,402 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:57,402 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:57,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:57,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:57,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:57,403 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:57,403 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:57,503 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:57,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:57,505 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:57,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:57,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:57,507 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:57,507 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:57,507 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:57,507 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:57,507 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,507 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:57,508 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:57,508 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:57,509 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,509 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:57,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:57,510 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:57,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:57,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:57,510 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:57,510 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:57,510 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:57,510 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:57,510 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:57,510 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:57,510 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:54:57,510 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-21 01:54:57,511 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:54:57,511 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:57,511 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:54:57,511 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:57,511 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:54:57,511 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:57,518 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/962563355d2c4759991cfb3595c322d3 2023-04-21 01:54:57,527 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/67d83b6c1d9341e38c0d44a9583431df 2023-04-21 01:54:57,537 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/d7b11eb3be664471b199add8b492fff2 2023-04-21 01:54:57,542 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/962563355d2c4759991cfb3595c322d3 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/962563355d2c4759991cfb3595c322d3 2023-04-21 01:54:57,556 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/962563355d2c4759991cfb3595c322d3, entries=150, sequenceid=136, filesize=11.8 K 2023-04-21 01:54:57,556 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/67d83b6c1d9341e38c0d44a9583431df as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/67d83b6c1d9341e38c0d44a9583431df 2023-04-21 01:54:57,560 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/67d83b6c1d9341e38c0d44a9583431df, entries=150, sequenceid=136, filesize=11.8 K 2023-04-21 01:54:57,561 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/d7b11eb3be664471b199add8b492fff2 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/d7b11eb3be664471b199add8b492fff2 2023-04-21 01:54:57,564 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/d7b11eb3be664471b199add8b492fff2, entries=150, sequenceid=136, filesize=11.8 K 2023-04-21 01:54:57,565 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 65bad43ef0e690f33e72a230ffd7ee4f in 55ms, sequenceid=136, compaction requested=true 2023-04-21 01:54:57,565 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:54:57,565 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:57,565 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:57,565 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:57,565 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,565 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:57,565 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:57,568 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:57,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:57,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36f841c4[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:57,570 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:57,570 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,571 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,571 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:57,571 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:57,571 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,571 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:57,572 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:57,573 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:57,573 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:57,573 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,573 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:57,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:57,575 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:57,575 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:57,575 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:57,575 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:57,575 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:57,577 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,577 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:57,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,577 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:57,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:57,577 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,579 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:57,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:57,582 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:57,582 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:57,582 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:57,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:57,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:57,582 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:57,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:57,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:57,583 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:57,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,583 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:57,583 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,583 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,683 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:57,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:57,685 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:57,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:57,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:57,686 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:57,687 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:57,687 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:57,687 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:57,687 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,687 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:57,688 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:57,688 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:57,689 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,689 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:57,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:57,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:57,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:57,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:57,690 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:57,690 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:57,690 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:57,691 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:57,691 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:57,691 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:57,691 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:54:57,691 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:54:57,691 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:57,691 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:57,691 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:57,691 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,691 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:57,691 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:57,692 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,692 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:57,693 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,693 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:57,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:57,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5015a9e8[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:57,694 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:57,694 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,695 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:57,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:57,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:57,696 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,696 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:57,698 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,698 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:57,698 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:57,698 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:57,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:57,700 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:57,700 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:57,700 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:57,700 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:57,700 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:57,702 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,702 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,702 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:57,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:57,702 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,704 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:57,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,709 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:57,709 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,709 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:57,709 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:57,709 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:57,709 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:57,709 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:57,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:57,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:57,709 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:57,709 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,709 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,709 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,709 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:57,709 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:57,710 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:57,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:54:57,743 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-21 01:54:57,744 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:54:57,744 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:57,744 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:54:57,744 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:57,744 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:54:57,744 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:57,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:57,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042157779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042097760, max timestamp=1682042097782Num uniques 859; 2023-04-21 01:54:57,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:54:57,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042097760, max timestamp=1682042097782Num uniques 859; 2023-04-21 01:54:57,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042097760, max timestamp=1682042097782Num uniques 859; 2023-04-21 01:54:57,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:54:57,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:54:57,810 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:57,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:57,812 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:57,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:57,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:57,813 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:57,813 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:57,813 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:57,813 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:57,814 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,814 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:57,821 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:57,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:57,821 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:57,821 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,822 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:57,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:57,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:57,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:57,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:57,822 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:57,822 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:57,823 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:57,823 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:57,823 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:57,823 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:57,823 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:54:57,823 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:54:57,823 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:57,823 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:57,823 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:57,823 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,823 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:57,823 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:57,824 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:57,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:57,824 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:57,825 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,825 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:57,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:57,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:57,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:57,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c325c2c[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:57,826 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:57,826 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:57,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042157823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,827 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:57,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:57,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:57,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:57,828 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,828 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:57,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52724 deadline: 1682042157825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,829 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:57,829 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:57,829 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,829 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:57,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042157827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:57,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:57,832 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:57,832 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:57,832 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:57,832 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:57,832 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:57,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:57,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:57,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:57,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:57,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:57,836 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,837 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:57,839 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:57,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:57,839 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:57,839 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:57,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:57,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,839 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:57,840 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:57,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:57,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:57,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:57,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:57,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042157884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:57,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042157928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:57,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52724 deadline: 1682042157929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:57,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042157932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,940 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:57,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:57,942 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:57,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:57,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:57,943 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:57,943 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:57,943 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:57,943 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:57,943 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,944 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:57,945 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,945 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:57,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:57,945 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,945 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:57,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:57,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:57,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:57,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:57,946 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:57,946 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:57,946 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:57,946 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:57,946 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:57,946 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:57,947 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:54:57,947 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:54:57,947 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:57,947 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:57,947 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:57,947 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,947 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:57,947 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:57,948 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,948 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:57,948 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,948 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:57,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:57,950 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:57,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@680bb04c[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:57,950 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,956 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,956 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:57,956 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:57,956 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:57,956 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,956 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:57,958 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:57,958 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,958 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:57,958 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:57,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:57,960 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:57,960 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:57,960 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:57,960 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:57,960 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:57,962 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,962 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,962 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:57,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:57,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:57,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:57,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:57,963 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,964 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,966 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,966 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:57,966 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:57,966 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:57,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:57,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:57,966 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:57,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:57,966 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:57,967 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:57,966 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:57,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:57,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:57,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:57,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,967 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:57,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:57,967 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:57,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,067 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:58,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:58,069 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:58,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:58,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:58,070 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:58,070 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:58,070 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:58,071 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:58,071 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,071 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,073 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:58,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:58,073 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,073 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:58,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:58,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:58,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:58,074 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:58,074 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:58,075 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:58,075 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:58,075 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:58,075 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:58,075 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:54:58,075 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:54:58,075 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:58,075 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:58,075 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:58,075 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,075 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:58,075 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:58,076 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,076 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:58,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:58,077 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,077 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:58,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:58,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:58,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:58,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:58,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c092ae8[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:58,079 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:58,079 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,080 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,080 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:58,080 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:58,080 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,080 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:58,080 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:58,082 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:58,082 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:58,082 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,082 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:58,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:58,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:58,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:58,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:58,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:58,084 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:58,084 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:58,084 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:58,084 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:58,084 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:58,087 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,087 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,087 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:58,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:58,087 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,088 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:58,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042158087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,091 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,091 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:58,091 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:58,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:58,091 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:58,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:58,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:58,091 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,091 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:58,091 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:58,091 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,091 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,091 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:58,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:58,091 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,091 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:58,091 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,092 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:58,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:58,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042158134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:58,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52724 deadline: 1682042158135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:58,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042158136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,151 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/040d73a3fd8941f0bfc43c789e927ddc 2023-04-21 01:54:58,168 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/3381dfa5347f47d788d0ea57a7a985b4 2023-04-21 01:54:58,192 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:58,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:58,194 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:58,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:58,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:58,195 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:58,195 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:58,195 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:58,195 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:58,195 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,195 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,199 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:58,199 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:58,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:58,199 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,200 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:58,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:58,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:58,201 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:58,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:58,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:58,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:58,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:58,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:58,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:58,201 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:58,201 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:54:58,201 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:54:58,201 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:58,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:58,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:58,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:58,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:58,204 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,204 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:58,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:58,204 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,204 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:58,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:58,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:58,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:58,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61ae8a68[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:58,206 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:58,206 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,207 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,207 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:58,207 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:58,207 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:58,207 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,207 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:58,208 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:58,208 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:58,208 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,208 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:58,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:58,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:58,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:58,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:58,211 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:58,211 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:58,211 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:58,211 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:58,211 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:58,212 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,212 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,212 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:58,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:58,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:58,213 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:58,214 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:58,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,218 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,218 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:58,218 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:58,218 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:58,218 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,218 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:58,218 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:58,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:58,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:58,218 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,218 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,218 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,218 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,218 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:58,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:58,218 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:58,218 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:58,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:58,319 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:58,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:58,321 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:58,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:58,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:58,322 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:58,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:58,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:58,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:58,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,323 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,323 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:58,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:58,324 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,324 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:58,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:58,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:58,325 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:58,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:58,325 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:58,325 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:54:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:58,325 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:54:58,325 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:58,327 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,327 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:58,327 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,327 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:58,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:58,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff3694d[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:58,329 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:58,329 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,330 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,330 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:58,330 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:58,330 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,330 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:58,330 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:58,333 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:58,333 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:58,333 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:58,333 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:58,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:58,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:58,335 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:58,336 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:58,336 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:58,336 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:58,336 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:58,337 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:58,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:58,337 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,337 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:58,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:58,338 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,339 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:58,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,341 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,341 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:58,341 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:58,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:58,341 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,341 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:58,341 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:58,341 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,341 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:58,341 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:58,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:58,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:58,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:58,341 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,342 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,342 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:58,342 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,342 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:58,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:58,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042158393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:58,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042158438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,442 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:58,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:58,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:58,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042158440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:58,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52724 deadline: 1682042158441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,444 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:58,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:58,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:58,445 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:58,445 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:58,445 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:58,445 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:58,446 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,446 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,447 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,447 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:58,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:58,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:58,447 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,447 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:58,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:58,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:58,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:58,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:58,448 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:58,448 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:58,448 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:58,449 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:58,449 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:58,449 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:58,449 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:54:58,449 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:54:58,449 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:58,449 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:58,449 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:58,449 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,449 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:58,449 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:58,451 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,451 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:58,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:58,452 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,452 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:58,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:58,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:58,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:58,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:58,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a4f2d2f[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:58,453 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:58,453 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,454 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,455 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:58,455 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:58,455 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:58,455 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,455 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:58,456 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:58,456 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:58,456 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,456 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:58,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:58,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:58,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:58,459 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:58,459 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:58,459 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:58,459 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:58,459 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:58,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:58,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:58,461 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,462 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:58,465 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:58,465 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:58,465 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:58,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:58,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:58,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:58,466 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:58,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,466 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:58,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:58,566 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:58,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:58,568 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:58,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:58,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:58,569 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:58,569 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:58,569 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:58,569 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:58,570 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,570 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,571 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,571 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:58,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:58,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:58,571 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,571 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:58,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:58,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,572 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:58,572 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:58,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:58,573 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:58,573 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:58,573 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:58,573 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:58,573 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:58,573 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:58,573 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:54:58,573 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:54:58,573 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:58,573 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:58,573 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:58,573 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,573 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:58,573 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:58,577 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:58,577 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:58,578 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,578 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:58,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:58,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fc402e4[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:58,579 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:58,579 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,581 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,581 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:58,581 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:58,581 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:58,581 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,581 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:58,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,582 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:58,583 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:58,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:58,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:58,583 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:58,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:58,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:58,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,584 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/6ff13d8675cc4240a7a9bc8c9089e81e 2023-04-21 01:54:58,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:58,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:58,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:58,587 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:58,588 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:58,588 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:58,588 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:58,588 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:58,589 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,589 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,590 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:58,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:58,590 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:58,590 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/040d73a3fd8941f0bfc43c789e927ddc as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/040d73a3fd8941f0bfc43c789e927ddc 2023-04-21 01:54:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:58,592 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:58,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:58,594 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,594 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,595 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,595 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,595 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,595 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:58,595 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:58,595 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:58,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:58,595 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:58,595 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:58,595 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:58,595 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/040d73a3fd8941f0bfc43c789e927ddc, entries=150, sequenceid=148, filesize=11.8 K 2023-04-21 01:54:58,595 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,595 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:58,595 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:58,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:58,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:58,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:58,596 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/3381dfa5347f47d788d0ea57a7a985b4 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/3381dfa5347f47d788d0ea57a7a985b4 2023-04-21 01:54:58,600 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/3381dfa5347f47d788d0ea57a7a985b4, entries=150, sequenceid=148, filesize=11.8 K 2023-04-21 01:54:58,600 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/6ff13d8675cc4240a7a9bc8c9089e81e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/6ff13d8675cc4240a7a9bc8c9089e81e 2023-04-21 01:54:58,603 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/6ff13d8675cc4240a7a9bc8c9089e81e, entries=150, sequenceid=148, filesize=11.8 K 2023-04-21 01:54:58,604 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 65bad43ef0e690f33e72a230ffd7ee4f in 861ms, sequenceid=148, compaction requested=true 2023-04-21 01:54:58,604 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:54:58,604 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:58,605 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-21 01:54:58,605 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-21 01:54:58,605 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:58,605 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:58,606 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71616 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-21 01:54:58,606 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71616 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-21 01:54:58,606 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/A is initiating minor compaction (all files) 2023-04-21 01:54:58,606 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/B is initiating minor compaction (all files) 2023-04-21 01:54:58,606 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/A in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:58,606 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/B in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:58,606 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/c0067f0b089e4a189c6241100fd87dee, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/b6e9b995e479405b82455e914f7c0ead, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/3dd5c55c600849889a96441959c441d2, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/014047aa59ec4fdba73ef1bcace58ea9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/962563355d2c4759991cfb3595c322d3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/040d73a3fd8941f0bfc43c789e927ddc] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=69.9 K 2023-04-21 01:54:58,606 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/ca3d6f55b9374776b92127793f0275cd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/c2211c36751c4aa59fa22ee75bb280e5, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/9568a1a25dc9402ea6ed1503f59edcfa, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/7cd5c498969c4695b9210c5d3de804d1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/67d83b6c1d9341e38c0d44a9583431df, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/3381dfa5347f47d788d0ea57a7a985b4] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=69.9 K 2023-04-21 01:54:58,607 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting c0067f0b089e4a189c6241100fd87dee, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1682042092729 2023-04-21 01:54:58,607 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting ca3d6f55b9374776b92127793f0275cd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1682042092729 2023-04-21 01:54:58,607 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting b6e9b995e479405b82455e914f7c0ead, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1682042092798 2023-04-21 01:54:58,607 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting c2211c36751c4aa59fa22ee75bb280e5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1682042092798 2023-04-21 01:54:58,607 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 3dd5c55c600849889a96441959c441d2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1682042094926 2023-04-21 01:54:58,608 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 9568a1a25dc9402ea6ed1503f59edcfa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1682042094926 2023-04-21 01:54:58,608 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 014047aa59ec4fdba73ef1bcace58ea9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682042095599 2023-04-21 01:54:58,608 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 7cd5c498969c4695b9210c5d3de804d1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682042095599 2023-04-21 01:54:58,608 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 962563355d2c4759991cfb3595c322d3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1682042096691 2023-04-21 01:54:58,608 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 67d83b6c1d9341e38c0d44a9583431df, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1682042096691 2023-04-21 01:54:58,609 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 040d73a3fd8941f0bfc43c789e927ddc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1682042097739 2023-04-21 01:54:58,609 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 3381dfa5347f47d788d0ea57a7a985b4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1682042097739 2023-04-21 01:54:58,621 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#B#compaction#252 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:58,621 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#A#compaction#253 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:58,637 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/9c2ba92e4efe4e0496a6e88bf6fc853a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/9c2ba92e4efe4e0496a6e88bf6fc853a 2023-04-21 01:54:58,641 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/B of 65bad43ef0e690f33e72a230ffd7ee4f into 9c2ba92e4efe4e0496a6e88bf6fc853a(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:58,641 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:54:58,641 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/B, priority=10, startTime=1682042098605; duration=0sec 2023-04-21 01:54:58,642 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:58,642 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-21 01:54:58,643 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71616 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-21 01:54:58,643 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/C is initiating minor compaction (all files) 2023-04-21 01:54:58,643 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/C in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:58,643 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/04933a20731f4bd29c1c513b40bf8e07, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/310a54e48c91461d8bac491a02a7a56a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/ef608552bce644c78fd173df6a833564, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/8ac454416e604d27bef5d30cf8296e03, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/d7b11eb3be664471b199add8b492fff2, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/6ff13d8675cc4240a7a9bc8c9089e81e] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=69.9 K 2023-04-21 01:54:58,644 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 04933a20731f4bd29c1c513b40bf8e07, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1682042092729 2023-04-21 01:54:58,644 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 310a54e48c91461d8bac491a02a7a56a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1682042092798 2023-04-21 01:54:58,644 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting ef608552bce644c78fd173df6a833564, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1682042094926 2023-04-21 01:54:58,645 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 8ac454416e604d27bef5d30cf8296e03, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682042095599 2023-04-21 01:54:58,645 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting d7b11eb3be664471b199add8b492fff2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1682042096691 2023-04-21 01:54:58,645 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 6ff13d8675cc4240a7a9bc8c9089e81e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1682042097739 2023-04-21 01:54:58,654 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#C#compaction#254 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:58,696 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:58,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:58,698 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:58,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:58,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:58,699 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:58,699 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:58,699 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:58,699 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:58,700 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,700 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,702 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,702 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:58,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:58,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:58,702 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,702 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:58,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:58,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:58,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:58,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:58,703 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:58,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:58,704 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:58,704 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:58,704 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:58,704 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:58,704 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:58,704 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:58,704 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:54:58,704 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-21 01:54:58,704 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:54:58,704 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:58,704 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:54:58,704 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:58,704 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:54:58,704 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:54:58,712 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/c0d51c47d69f41b3aa3bde6407d60a92 2023-04-21 01:54:58,730 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/340b07ec39ec443e8f9709b53b05a89d 2023-04-21 01:54:58,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:54:58,904 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:54:58,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:58,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042158930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,944 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:58,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042158943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:58,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042158944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:58,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:58,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52724 deadline: 1682042158945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,037 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/df19bee729864e1080e2e31093941a78 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/df19bee729864e1080e2e31093941a78 2023-04-21 01:54:59,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:59,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042159037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,042 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/A of 65bad43ef0e690f33e72a230ffd7ee4f into df19bee729864e1080e2e31093941a78(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:59,042 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:54:59,042 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/A, priority=10, startTime=1682042098604; duration=0sec 2023-04-21 01:54:59,042 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:59,062 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/e7f3325fa0904836bb38b0697fbce50b as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/e7f3325fa0904836bb38b0697fbce50b 2023-04-21 01:54:59,066 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/C of 65bad43ef0e690f33e72a230ffd7ee4f into e7f3325fa0904836bb38b0697fbce50b(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:59,066 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:54:59,066 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/C, priority=10, startTime=1682042098605; duration=0sec 2023-04-21 01:54:59,066 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:59,140 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/77258a14435f4ec98128954424ba5af3 2023-04-21 01:54:59,145 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/c0d51c47d69f41b3aa3bde6407d60a92 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/c0d51c47d69f41b3aa3bde6407d60a92 2023-04-21 01:54:59,149 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/c0d51c47d69f41b3aa3bde6407d60a92, entries=150, sequenceid=185, filesize=11.8 K 2023-04-21 01:54:59,150 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/340b07ec39ec443e8f9709b53b05a89d as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/340b07ec39ec443e8f9709b53b05a89d 2023-04-21 01:54:59,153 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/340b07ec39ec443e8f9709b53b05a89d, entries=150, sequenceid=185, filesize=11.8 K 2023-04-21 01:54:59,154 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/77258a14435f4ec98128954424ba5af3 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/77258a14435f4ec98128954424ba5af3 2023-04-21 01:54:59,157 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/77258a14435f4ec98128954424ba5af3, entries=150, sequenceid=185, filesize=11.8 K 2023-04-21 01:54:59,158 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 65bad43ef0e690f33e72a230ffd7ee4f in 454ms, sequenceid=185, compaction requested=false 2023-04-21 01:54:59,158 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:54:59,158 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:59,158 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:59,158 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:59,158 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,158 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:59,158 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:59,160 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:59,160 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:59,160 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,160 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:59,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:59,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a2c34be[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:59,162 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:59,162 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,163 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,163 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:59,163 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:59,163 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:59,163 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,163 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:59,164 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,164 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:59,164 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:59,164 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:59,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:59,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:59,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:59,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:59,167 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:59,167 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:59,167 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:59,167 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:59,167 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:59,169 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,169 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:59,169 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:59,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:59,170 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:59,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:59,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,171 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,174 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:59,174 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:59,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:59,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:59,174 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,174 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,174 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,174 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:59,174 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:59,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:59,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:59,175 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:59,175 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:59,174 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:59,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:59,174 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,175 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,175 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,175 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:54:59,242 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-21 01:54:59,242 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:54:59,242 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:59,242 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:54:59,242 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:59,242 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:54:59,242 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:59,275 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:59,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:59,277 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:59,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:59,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:59,278 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:59,278 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:59,278 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:59,278 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:59,279 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,279 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,280 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:59,280 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:59,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:59,280 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,280 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:59,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:59,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:59,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:59,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:59,281 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:59,281 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:59,281 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:59,282 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:59,282 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:54:59,282 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:54:59,282 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:59,284 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,284 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:59,284 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,284 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:59,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:59,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ee18fb7[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:59,285 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:59,286 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,290 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,291 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:59,291 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:59,291 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:59,291 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,291 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:59,292 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,292 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:59,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:59,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:59,292 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:59,292 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:59,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:59,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:59,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:59,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042159290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:59,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:59,294 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:59,295 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:59,295 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:59,295 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:59,295 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:59,296 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,296 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,296 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:59,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,296 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:59,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,298 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:59,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,299 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,299 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,299 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,299 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,299 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,299 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:59,300 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:59,299 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:59,300 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:59,299 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:59,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:59,300 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:59,300 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:59,300 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,300 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:59,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:59,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:59,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:59,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:59,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042159394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,400 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:59,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:59,402 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:59,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:59,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:59,404 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:59,404 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:59,404 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:59,404 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:59,404 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,404 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,407 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,407 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:59,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:59,407 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,407 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:59,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:59,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:59,408 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:59,409 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:59,409 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:54:59,409 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:54:59,409 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:59,411 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,411 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:59,411 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,411 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:59,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:59,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d64f111[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:59,412 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:59,413 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,414 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,414 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:59,414 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:59,414 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,414 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:59,414 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:59,418 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:59,418 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,418 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:59,419 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:59,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:59,421 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:59,421 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:59,421 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:59,421 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:59,421 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:59,422 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,423 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:59,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:59,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,423 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:59,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:59,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:59,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,424 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,425 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,426 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,426 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,426 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,427 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,427 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:59,427 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:59,427 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:59,427 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,427 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:59,427 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:59,427 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:59,427 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,427 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:59,427 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:59,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:59,527 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:59,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:59,529 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:59,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:59,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:59,530 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:59,530 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:59,531 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:59,531 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:59,532 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,532 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,533 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,533 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:59,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:59,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:59,533 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,533 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:59,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:59,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,534 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:59,534 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:59,534 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:59,534 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:59,534 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:59,535 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:59,535 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:59,535 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:59,535 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:59,535 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:54:59,535 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:54:59,535 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:59,535 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:59,535 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:59,535 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,535 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:59,535 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:59,543 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,543 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:59,543 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,543 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:59,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:59,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6298197b[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:59,545 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:59,545 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,546 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,546 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:59,546 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:59,546 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:59,546 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,546 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:59,547 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:59,547 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,547 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:59,547 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:59,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:59,551 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:59,551 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:59,551 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:59,551 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:59,551 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:59,553 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,553 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,553 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:59,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:59,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:59,553 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,555 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,558 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:59,558 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:59,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:59,558 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:59,558 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:59,558 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:59,558 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,558 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:59,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:59,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:59,559 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,559 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:59,559 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:59,559 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,559 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,559 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,559 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:59,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042159597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,650 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/725934dcc21649619e4b80953aeb940f 2023-04-21 01:54:59,659 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:59,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:59,660 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/abf5e1d39cf24e338a2d143f26c72b90 2023-04-21 01:54:59,662 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:59,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:59,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:59,663 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:59,663 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:59,663 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:59,663 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:59,664 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,664 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,665 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:59,665 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:59,666 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,666 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:59,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:59,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,667 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:59,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:59,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:59,670 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:59,670 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:59,670 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:59,671 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:59,671 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:59,671 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:59,671 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:54:59,671 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:54:59,671 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:59,671 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:54:59,671 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:54:59,671 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,671 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:54:59,671 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:54:59,673 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,673 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:59,673 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,674 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:54:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:59,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:54:59,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@196a4852[Count = 0] remaining members to acquire global barrier 2023-04-21 01:54:59,675 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:54:59,675 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,676 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/f763d1b4ce9b465b86d794854eb8d611 2023-04-21 01:54:59,677 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,677 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:54:59,679 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,679 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:54:59,679 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:54:59,679 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:54:59,682 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/725934dcc21649619e4b80953aeb940f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/725934dcc21649619e4b80953aeb940f 2023-04-21 01:54:59,682 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:54:59,682 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,682 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:59,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:59,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:59,682 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:54:59,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:59,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:59,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:59,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:54:59,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:54:59,685 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:54:59,685 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:54:59,685 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:54:59,685 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:54:59,685 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:54:59,686 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/725934dcc21649619e4b80953aeb940f, entries=150, sequenceid=199, filesize=11.8 K 2023-04-21 01:54:59,686 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,687 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:59,687 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:54:59,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:54:59,687 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/abf5e1d39cf24e338a2d143f26c72b90 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/abf5e1d39cf24e338a2d143f26c72b90 2023-04-21 01:54:59,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:59,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:59,687 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:54:59,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:54:59,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,688 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,691 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/abf5e1d39cf24e338a2d143f26c72b90, entries=150, sequenceid=199, filesize=11.8 K 2023-04-21 01:54:59,691 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/f763d1b4ce9b465b86d794854eb8d611 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/f763d1b4ce9b465b86d794854eb8d611 2023-04-21 01:54:59,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:54:59,696 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/f763d1b4ce9b465b86d794854eb8d611, entries=150, sequenceid=199, filesize=11.8 K 2023-04-21 01:54:59,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:54:59,697 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 65bad43ef0e690f33e72a230ffd7ee4f in 455ms, sequenceid=199, compaction requested=true 2023-04-21 01:54:59,697 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:54:59,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,697 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:59,697 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:54:59,697 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:59,697 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-21 01:54:59,697 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:59,699 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:59,699 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/A is initiating minor compaction (all files) 2023-04-21 01:54:59,699 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/A in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:59,699 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/df19bee729864e1080e2e31093941a78, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/c0d51c47d69f41b3aa3bde6407d60a92, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/725934dcc21649619e4b80953aeb940f] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=35.5 K 2023-04-21 01:54:59,700 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,700 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,700 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,700 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,700 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:59,700 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:54:59,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:59,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:59,700 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,700 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:54:59,700 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,700 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:54:59,700 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:54:59,700 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:54:59,701 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:59,701 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/B is initiating minor compaction (all files) 2023-04-21 01:54:59,701 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:54:59,701 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:54:59,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:54:59,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:54:59,701 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/B in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:59,702 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/9c2ba92e4efe4e0496a6e88bf6fc853a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/340b07ec39ec443e8f9709b53b05a89d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/abf5e1d39cf24e338a2d143f26c72b90] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=35.5 K 2023-04-21 01:54:59,702 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting df19bee729864e1080e2e31093941a78, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1682042097739 2023-04-21 01:54:59,703 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 9c2ba92e4efe4e0496a6e88bf6fc853a, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1682042097739 2023-04-21 01:54:59,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:54:59,703 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting c0d51c47d69f41b3aa3bde6407d60a92, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1682042097820 2023-04-21 01:54:59,704 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 340b07ec39ec443e8f9709b53b05a89d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1682042097820 2023-04-21 01:54:59,704 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 725934dcc21649619e4b80953aeb940f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682042098922 2023-04-21 01:54:59,704 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting abf5e1d39cf24e338a2d143f26c72b90, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682042098922 2023-04-21 01:54:59,723 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#A#compaction#261 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:59,723 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#B#compaction#262 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:59,737 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/6b569012bdb94b059ce154b348f30b55 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/6b569012bdb94b059ce154b348f30b55 2023-04-21 01:54:59,741 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/B of 65bad43ef0e690f33e72a230ffd7ee4f into 6b569012bdb94b059ce154b348f30b55(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:54:59,741 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:54:59,741 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/B, priority=13, startTime=1682042099697; duration=0sec 2023-04-21 01:54:59,742 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:54:59,742 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:54:59,742 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:54:59,743 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/C is initiating minor compaction (all files) 2023-04-21 01:54:59,743 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/C in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:59,743 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/e7f3325fa0904836bb38b0697fbce50b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/77258a14435f4ec98128954424ba5af3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/f763d1b4ce9b465b86d794854eb8d611] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=35.5 K 2023-04-21 01:54:59,743 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting e7f3325fa0904836bb38b0697fbce50b, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1682042097739 2023-04-21 01:54:59,743 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 77258a14435f4ec98128954424ba5af3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1682042097820 2023-04-21 01:54:59,744 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting f763d1b4ce9b465b86d794854eb8d611, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682042098922 2023-04-21 01:54:59,750 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#C#compaction#263 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:54:59,801 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:54:59,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:54:59,803 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:54:59,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:54:59,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:54:59,804 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:54:59,804 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:59,805 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:54:59,805 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:54:59,805 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,805 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,806 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,806 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:54:59,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:54:59,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:54:59,807 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,807 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:54:59,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:54:59,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:54:59,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:54:59,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:54:59,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:54:59,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:54:59,808 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:54:59,808 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:54:59,808 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:54:59,808 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:54:59,808 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:54:59,808 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:54:59,808 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:54:59,808 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-21 01:54:59,809 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:54:59,809 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:59,809 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:54:59,809 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:59,809 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:54:59,809 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:54:59,822 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/823ca5dc7eb243c49e74d44c60711cf3 2023-04-21 01:54:59,833 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/e40ce5703f3a4e87a540f86f841ad4c1 2023-04-21 01:54:59,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:54:59,903 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:54:59,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:59,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042159923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,954 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:59,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042159952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:59,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042159955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:54:59,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:54:59,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52724 deadline: 1682042159955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:00,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042160026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,141 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/9caa7e1f78f248e381e0d9d7ab430836 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/9caa7e1f78f248e381e0d9d7ab430836 2023-04-21 01:55:00,145 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/A of 65bad43ef0e690f33e72a230ffd7ee4f into 9caa7e1f78f248e381e0d9d7ab430836(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:00,146 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:00,146 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/A, priority=13, startTime=1682042099697; duration=0sec 2023-04-21 01:55:00,146 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:00,162 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/8e77c9befa53459580decfd699a6b091 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/8e77c9befa53459580decfd699a6b091 2023-04-21 01:55:00,166 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/C of 65bad43ef0e690f33e72a230ffd7ee4f into 8e77c9befa53459580decfd699a6b091(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:00,166 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:00,166 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/C, priority=13, startTime=1682042099697; duration=0sec 2023-04-21 01:55:00,166 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:00,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:00,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52752 deadline: 1682042160228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,244 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/f3060cac2bab492e9290bb96fa257907 2023-04-21 01:55:00,248 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/823ca5dc7eb243c49e74d44c60711cf3 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/823ca5dc7eb243c49e74d44c60711cf3 2023-04-21 01:55:00,253 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/823ca5dc7eb243c49e74d44c60711cf3, entries=150, sequenceid=220, filesize=11.8 K 2023-04-21 01:55:00,254 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/e40ce5703f3a4e87a540f86f841ad4c1 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/e40ce5703f3a4e87a540f86f841ad4c1 2023-04-21 01:55:00,258 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/e40ce5703f3a4e87a540f86f841ad4c1, entries=150, sequenceid=220, filesize=11.8 K 2023-04-21 01:55:00,258 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/f3060cac2bab492e9290bb96fa257907 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/f3060cac2bab492e9290bb96fa257907 2023-04-21 01:55:00,262 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/f3060cac2bab492e9290bb96fa257907, entries=150, sequenceid=220, filesize=11.8 K 2023-04-21 01:55:00,263 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 65bad43ef0e690f33e72a230ffd7ee4f in 455ms, sequenceid=220, compaction requested=false 2023-04-21 01:55:00,263 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:00,263 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:00,263 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:00,263 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:00,263 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,263 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:00,263 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:00,265 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,265 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:00,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:00,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:00,266 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:00,266 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:00,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:00,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:00,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:00,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:00,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:00,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b94f0ff[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:00,267 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:00,268 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:00,269 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:00,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:00,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:00,269 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:00,269 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:00,269 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:00,269 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,270 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:00,270 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:00,270 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,270 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:00,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:00,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:00,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:00,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:00,274 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:00,274 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:00,274 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:00,274 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:00,274 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:00,283 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:00,283 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:00,283 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:00,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:00,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:00,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:00,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:00,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:00,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:00,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:00,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:00,284 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:00,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:00,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:00,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:00,285 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,287 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,287 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:00,287 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:00,287 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:00,287 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:00,287 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:00,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:00,287 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:00,287 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:00,287 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:00,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:00,287 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:00,288 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:00,288 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:00,288 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:00,288 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:00,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:00,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:00,388 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:00,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:00,390 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:00,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:00,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:00,391 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:00,392 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:00,392 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:00,392 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:00,392 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:00,392 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:00,393 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,393 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:00,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:00,394 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,394 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:00,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:00,395 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:00,395 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:00,395 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:00,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:00,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:00,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:00,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:00,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:00,396 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:00,396 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:00,396 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-21 01:55:00,396 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:55:00,396 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:00,396 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:55:00,396 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:00,396 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:55:00,396 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:00,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:55:00,533 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:00,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:00,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042160586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:00,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042160688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,805 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/311fd1aa775d4fe885096f374ef33c1b 2023-04-21 01:55:00,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:00,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042160891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,956 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:00,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52802 deadline: 1682042160955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:00,957 DEBUG [Thread-1862] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:55:01,196 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:01,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042161195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,219 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/f1e87e89c13646f18576173bd9d45724 2023-04-21 01:55:01,628 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/8a5d066dfaf74159a57629cbf9bd7656 2023-04-21 01:55:01,633 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/311fd1aa775d4fe885096f374ef33c1b as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/311fd1aa775d4fe885096f374ef33c1b 2023-04-21 01:55:01,636 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/311fd1aa775d4fe885096f374ef33c1b, entries=150, sequenceid=234, filesize=11.8 K 2023-04-21 01:55:01,637 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/f1e87e89c13646f18576173bd9d45724 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/f1e87e89c13646f18576173bd9d45724 2023-04-21 01:55:01,640 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/f1e87e89c13646f18576173bd9d45724, entries=150, sequenceid=234, filesize=11.8 K 2023-04-21 01:55:01,641 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/8a5d066dfaf74159a57629cbf9bd7656 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/8a5d066dfaf74159a57629cbf9bd7656 2023-04-21 01:55:01,645 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/8a5d066dfaf74159a57629cbf9bd7656, entries=150, sequenceid=234, filesize=11.8 K 2023-04-21 01:55:01,646 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 65bad43ef0e690f33e72a230ffd7ee4f in 1249ms, sequenceid=234, compaction requested=true 2023-04-21 01:55:01,646 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:01,646 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:01,646 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:01,646 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:01,646 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,646 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:01,646 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:01,647 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,647 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:01,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:01,648 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,648 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:01,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:01,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:01,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11927ae5[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:01,650 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:01,650 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,652 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,653 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:01,653 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:01,653 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,653 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:01,653 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:01,654 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,654 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:01,654 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:01,654 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:01,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:01,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:01,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:01,657 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:01,657 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:01,657 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:01,657 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:01,657 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:01,659 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,659 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:01,659 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:01,660 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:01,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:01,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:01,662 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:01,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:01,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,664 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,664 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:01,664 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:01,664 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:01,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:01,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:01,664 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:01,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:01,664 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:01,664 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:01,664 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:01,665 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:01,665 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:01,665 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,665 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,665 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,665 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:01,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:55:01,703 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-21 01:55:01,703 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:55:01,703 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:01,703 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:55:01,703 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:01,703 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:55:01,703 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:01,711 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/04130290fbc743b496ea61d17f2f8f98 2023-04-21 01:55:01,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:01,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042161728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,765 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:01,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:01,767 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:01,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:01,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:01,768 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:01,768 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:01,768 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:01,768 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:01,768 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,768 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:01,772 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,772 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:01,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:01,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:01,773 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,773 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:01,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:01,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:01,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:01,773 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:01,774 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:01,774 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:01,774 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:01,774 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:01,774 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:01,774 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:01,774 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:01,774 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:01,774 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:01,774 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:01,774 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:01,774 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:01,774 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:01,774 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,774 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:01,774 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:01,776 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:01,776 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:01,776 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,776 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:01,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:01,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:01,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:01,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33e3ea6d[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:01,777 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:01,777 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,778 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,779 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,779 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:01,779 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:01,779 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:01,779 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:01,780 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,780 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:01,780 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:01,780 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:01,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:01,782 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:01,782 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:01,782 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:01,782 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:01,782 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:01,784 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,784 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:01,784 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:01,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:01,785 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:01,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:01,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:01,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:01,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:01,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,786 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:01,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:01,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,788 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,788 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:01,788 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:01,789 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:01,789 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:01,789 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:01,789 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:01,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:01,789 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:01,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:01,789 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:01,789 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,789 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:01,789 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:01,789 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,789 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:01,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042161832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,889 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:01,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:01,891 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:01,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:01,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:01,892 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:01,892 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:01,892 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:01,893 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:01,893 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,893 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:01,894 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,894 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:01,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:01,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:01,894 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,894 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:01,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:01,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:01,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:01,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:01,896 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:01,896 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:01,896 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:01,896 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:01,896 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:01,896 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:01,896 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:01,896 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:01,896 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:01,896 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:01,896 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:01,896 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,896 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:01,896 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:01,898 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:01,898 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:01,898 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,898 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:01,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:01,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78001c9b[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:01,899 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:01,900 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,901 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,902 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:01,902 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:01,902 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:01,902 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,902 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:01,903 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:01,903 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:01,903 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,903 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:01,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:01,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:01,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:01,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:01,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:01,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:01,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:01,906 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:01,906 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:01,906 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:01,906 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:01,906 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:01,907 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,907 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,907 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:01,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:01,908 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:01,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:01,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:01,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:01,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:01,909 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:01,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:01,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,914 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,914 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:01,914 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:01,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:01,914 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:01,914 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:01,914 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:01,915 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:01,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:01,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:01,914 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:01,914 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,915 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:01,915 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:01,915 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,915 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:01,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:01,915 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:01,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:01,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:01,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042161961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,965 DEBUG [Thread-1866] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:55:01,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:01,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042161965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,968 DEBUG [Thread-1864] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:55:01,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:01,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52724 deadline: 1682042161971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:01,973 DEBUG [Thread-1860] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:55:02,015 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:02,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:02,017 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:02,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:02,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:02,019 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:02,019 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:02,019 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:02,019 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:02,019 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,019 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,020 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,020 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:02,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:02,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:02,021 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,021 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:02,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:02,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:02,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:02,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:02,022 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:02,022 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:02,022 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:02,022 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:02,022 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:02,022 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:02,022 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:02,022 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:02,022 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:02,022 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:02,023 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:02,023 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,023 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:02,023 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:02,024 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,024 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:02,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:02,024 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,024 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:02,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:02,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:02,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:02,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:02,026 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:02,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@244f186c[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:02,026 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,027 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,027 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:02,027 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:02,027 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:02,027 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,027 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:02,029 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:02,030 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:02,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,030 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:02,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:02,032 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:02,032 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:02,032 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:02,032 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:02,032 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:02,034 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,034 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,034 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:02,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:02,034 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:02,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,036 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:02,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:02,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042162036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,038 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:02,038 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:02,038 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:02,038 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:02,038 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:02,038 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:02,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:02,038 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,038 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,038 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,039 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,039 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,039 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,039 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:02,039 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:02,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:02,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:02,122 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/dbb3054afe19435c88bf424d63b1de9e 2023-04-21 01:55:02,139 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:02,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:02,141 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:02,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:02,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:02,142 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:02,142 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:02,142 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:02,142 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:02,142 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,142 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,143 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,143 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:02,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:02,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:02,144 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,144 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:02,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:02,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:02,145 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:02,145 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:02,145 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:02,145 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:02,145 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:02,145 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:02,145 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:02,145 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:02,145 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:02,145 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:02,146 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:02,146 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:02,146 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:02,146 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,146 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:02,146 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:02,148 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,148 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:02,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:02,148 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:02,148 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:02,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:02,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:02,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:02,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26881b52[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:02,150 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:02,150 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,151 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,151 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:02,151 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:02,151 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,151 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:02,151 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:02,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:02,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:02,152 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:02,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:02,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:02,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:02,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:02,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:02,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:02,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:02,155 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:02,155 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:02,155 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:02,155 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:02,155 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:02,156 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,156 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:02,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,156 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:02,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,157 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:02,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,158 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:02,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,161 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,161 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:02,161 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:02,161 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:02,161 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:02,161 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:02,162 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:02,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:02,161 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,162 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,162 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,162 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,162 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,162 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:02,162 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:02,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:02,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:02,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:02,262 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:02,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:02,264 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:02,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:02,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:02,267 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:02,267 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:02,267 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:02,267 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:02,268 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,268 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,269 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:02,269 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:02,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:02,269 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,269 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:02,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:02,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:02,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:02,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:02,270 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:02,270 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:02,270 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:02,270 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:02,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:02,271 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:02,271 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:02,271 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:02,271 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:02,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:02,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:02,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:02,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:02,272 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:02,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:02,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:02,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:02,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a142d6c[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:02,274 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:02,274 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,275 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,275 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:02,275 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:02,275 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:02,275 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,275 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:02,277 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,277 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:02,278 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:02,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:02,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:02,278 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:02,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:02,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:02,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:02,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:02,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:02,280 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:02,280 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:02,280 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:02,280 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:02,280 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:02,281 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,281 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:02,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:02,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,281 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:02,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:02,282 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:02,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:02,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:02,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,283 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,285 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,285 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,285 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,285 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:02,285 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:02,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:02,285 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:02,285 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:02,285 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:02,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:02,285 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:02,285 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,285 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:02,285 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:02,285 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,285 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:02,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:02,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:02,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042162341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,385 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:02,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:02,387 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:02,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:02,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:02,389 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:02,389 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:02,389 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:02,389 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:02,389 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,389 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,394 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:02,394 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:02,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:02,394 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,394 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:02,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:02,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,395 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:02,395 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:02,395 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:02,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:02,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:02,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:02,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:02,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:02,395 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:02,396 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:02,396 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:02,396 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:02,396 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:02,396 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:02,396 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,396 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:02,396 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:02,397 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,397 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:02,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:02,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:02,398 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,398 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:02,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:02,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:02,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:02,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72080c71[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:02,399 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:02,399 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,400 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,400 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:02,400 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:02,400 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:02,400 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,401 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:02,402 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,402 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:02,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:02,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:02,402 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:02,402 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:02,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:02,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:02,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:02,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:02,404 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:02,405 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:02,405 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:02,405 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:02,405 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:02,407 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,407 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:02,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:02,408 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:02,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:02,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,408 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:02,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:02,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:02,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,409 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:02,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,411 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:02,411 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:02,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:02,411 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:02,411 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:02,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:02,411 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:02,411 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,411 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:02,412 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,412 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:02,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:02,412 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,412 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:02,412 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:02,412 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,412 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,512 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:02,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:02,514 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:02,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:02,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:02,515 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:02,516 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:02,516 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:02,516 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:02,516 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,516 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,517 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,517 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:02,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:02,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:02,518 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,518 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:02,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:02,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:02,519 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:02,519 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:02,519 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:02,519 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:02,519 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:02,519 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:02,519 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:02,519 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:02,519 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:02,519 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:02,519 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:02,519 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:02,519 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:02,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:02,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:02,521 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,521 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:02,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:02,521 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,521 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:02,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:02,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:02,523 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:02,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e58e624[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:02,523 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,525 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,525 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:02,525 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:02,525 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:02,525 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,525 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:02,527 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:02,527 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:02,527 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,527 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:02,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:02,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:02,529 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:02,529 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:02,529 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:02,529 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:02,529 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:02,530 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,531 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:02,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,531 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:02,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:02,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:02,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:02,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,532 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:02,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,533 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/2f424c26af1c40efb9a34ca8f0c311cc 2023-04-21 01:55:02,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:02,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,534 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:02,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:02,537 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:02,537 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:02,537 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,537 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,537 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:02,537 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:02,537 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,537 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:02,537 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,537 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:02,537 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:02,537 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:02,538 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/04130290fbc743b496ea61d17f2f8f98 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/04130290fbc743b496ea61d17f2f8f98 2023-04-21 01:55:02,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:02,538 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:02,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:02,538 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:02,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:02,541 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/04130290fbc743b496ea61d17f2f8f98, entries=150, sequenceid=255, filesize=11.8 K 2023-04-21 01:55:02,542 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/dbb3054afe19435c88bf424d63b1de9e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/dbb3054afe19435c88bf424d63b1de9e 2023-04-21 01:55:02,546 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/dbb3054afe19435c88bf424d63b1de9e, entries=150, sequenceid=255, filesize=11.8 K 2023-04-21 01:55:02,547 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/2f424c26af1c40efb9a34ca8f0c311cc as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/2f424c26af1c40efb9a34ca8f0c311cc 2023-04-21 01:55:02,550 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/2f424c26af1c40efb9a34ca8f0c311cc, entries=150, sequenceid=255, filesize=11.8 K 2023-04-21 01:55:02,551 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 65bad43ef0e690f33e72a230ffd7ee4f in 848ms, sequenceid=255, compaction requested=true 2023-04-21 01:55:02,551 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:02,551 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:02,551 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:55:02,551 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:55:02,551 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:02,551 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:02,552 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:55:02,552 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:55:02,552 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/A is initiating minor compaction (all files) 2023-04-21 01:55:02,553 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/B is initiating minor compaction (all files) 2023-04-21 01:55:02,553 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/A in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:02,553 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/B in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:02,553 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/9caa7e1f78f248e381e0d9d7ab430836, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/823ca5dc7eb243c49e74d44c60711cf3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/311fd1aa775d4fe885096f374ef33c1b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/04130290fbc743b496ea61d17f2f8f98] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=47.3 K 2023-04-21 01:55:02,553 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/6b569012bdb94b059ce154b348f30b55, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/e40ce5703f3a4e87a540f86f841ad4c1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/f1e87e89c13646f18576173bd9d45724, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/dbb3054afe19435c88bf424d63b1de9e] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=47.3 K 2023-04-21 01:55:02,553 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 9caa7e1f78f248e381e0d9d7ab430836, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682042098922 2023-04-21 01:55:02,553 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 6b569012bdb94b059ce154b348f30b55, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682042098922 2023-04-21 01:55:02,553 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 823ca5dc7eb243c49e74d44c60711cf3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1682042099256 2023-04-21 01:55:02,553 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting e40ce5703f3a4e87a540f86f841ad4c1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1682042099256 2023-04-21 01:55:02,554 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 311fd1aa775d4fe885096f374ef33c1b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1682042099913 2023-04-21 01:55:02,554 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting f1e87e89c13646f18576173bd9d45724, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1682042099913 2023-04-21 01:55:02,554 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 04130290fbc743b496ea61d17f2f8f98, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682042100562 2023-04-21 01:55:02,554 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting dbb3054afe19435c88bf424d63b1de9e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682042100562 2023-04-21 01:55:02,564 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#B#compaction#273 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:02,564 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#A#compaction#274 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:02,638 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:02,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:02,640 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:02,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:02,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:02,641 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:02,641 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:02,641 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:02,641 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:02,642 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,642 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,644 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,644 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:02,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:02,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:02,644 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,644 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:02,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:02,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:02,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:02,645 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:02,645 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:02,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:02,645 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:02,645 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:02,645 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:02,646 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:02,646 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:02,646 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:02,646 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:02,646 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-21 01:55:02,646 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:55:02,646 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:02,646 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:55:02,646 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:02,647 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:55:02,647 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:02,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:55:02,844 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:02,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042102879, max timestamp=1682042102887Num uniques 859; 2023-04-21 01:55:02,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042102879, max timestamp=1682042102887Num uniques 859; 2023-04-21 01:55:02,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042102879, max timestamp=1682042102887Num uniques 859; 2023-04-21 01:55:02,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:02,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:02,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:02,949 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:02,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042162946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:02,976 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/51cda5dcdedf4ef7ae3b5f9d2b6f4145 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/51cda5dcdedf4ef7ae3b5f9d2b6f4145 2023-04-21 01:55:02,976 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/8aee80d3205b48dfa766bf162416878a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/8aee80d3205b48dfa766bf162416878a 2023-04-21 01:55:02,980 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/B of 65bad43ef0e690f33e72a230ffd7ee4f into 51cda5dcdedf4ef7ae3b5f9d2b6f4145(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:02,980 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:02,980 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/B, priority=12, startTime=1682042102551; duration=0sec 2023-04-21 01:55:02,980 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:02,980 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:55:02,981 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/A of 65bad43ef0e690f33e72a230ffd7ee4f into 8aee80d3205b48dfa766bf162416878a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:02,981 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:02,982 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/A, priority=12, startTime=1682042102551; duration=0sec 2023-04-21 01:55:02,982 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:55:02,982 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:02,982 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/C is initiating minor compaction (all files) 2023-04-21 01:55:02,982 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/C in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:02,982 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/8e77c9befa53459580decfd699a6b091, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/f3060cac2bab492e9290bb96fa257907, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/8a5d066dfaf74159a57629cbf9bd7656, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/2f424c26af1c40efb9a34ca8f0c311cc] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=47.3 K 2023-04-21 01:55:02,982 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 8e77c9befa53459580decfd699a6b091, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682042098922 2023-04-21 01:55:02,983 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting f3060cac2bab492e9290bb96fa257907, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1682042099256 2023-04-21 01:55:02,983 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 8a5d066dfaf74159a57629cbf9bd7656, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1682042099913 2023-04-21 01:55:02,983 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 2f424c26af1c40efb9a34ca8f0c311cc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682042100562 2023-04-21 01:55:02,991 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#C#compaction#276 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:03,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:03,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042163050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,053 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/008f70626533407db9586fb727ed1e24 2023-04-21 01:55:03,063 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/1a58373ef20741a7b40681733ec19f2c 2023-04-21 01:55:03,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:03,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042163251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,400 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/6147ebe5828e418499e07714cdfd153c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/6147ebe5828e418499e07714cdfd153c 2023-04-21 01:55:03,404 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/C of 65bad43ef0e690f33e72a230ffd7ee4f into 6147ebe5828e418499e07714cdfd153c(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:03,404 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:03,404 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/C, priority=12, startTime=1682042102551; duration=0sec 2023-04-21 01:55:03,404 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:03,474 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/a3094ddd28854e1897129b10ac8628b0 2023-04-21 01:55:03,479 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/008f70626533407db9586fb727ed1e24 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/008f70626533407db9586fb727ed1e24 2023-04-21 01:55:03,482 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/008f70626533407db9586fb727ed1e24, entries=100, sequenceid=266, filesize=9.5 K 2023-04-21 01:55:03,483 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/1a58373ef20741a7b40681733ec19f2c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/1a58373ef20741a7b40681733ec19f2c 2023-04-21 01:55:03,487 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/1a58373ef20741a7b40681733ec19f2c, entries=100, sequenceid=266, filesize=9.5 K 2023-04-21 01:55:03,488 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/a3094ddd28854e1897129b10ac8628b0 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a3094ddd28854e1897129b10ac8628b0 2023-04-21 01:55:03,491 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a3094ddd28854e1897129b10ac8628b0, entries=100, sequenceid=266, filesize=9.5 K 2023-04-21 01:55:03,491 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 65bad43ef0e690f33e72a230ffd7ee4f in 845ms, sequenceid=266, compaction requested=false 2023-04-21 01:55:03,492 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:03,492 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:03,493 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:03,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:03,493 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:03,494 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,494 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:03,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:03,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@486c3a36[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:03,495 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:03,495 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,496 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,496 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:03,496 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:03,496 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:03,497 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,497 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:03,498 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,498 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:03,498 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:03,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:03,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:03,498 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:03,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:03,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:03,500 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:03,500 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:03,500 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:03,500 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:03,501 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:03,503 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,503 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:03,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,503 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:03,504 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,505 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,507 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,507 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:03,507 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:03,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:03,507 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:03,507 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:03,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:03,507 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:03,507 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:03,507 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:03,507 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:03,507 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,507 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:03,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:03,507 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,508 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,508 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:55:03,557 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-21 01:55:03,558 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:55:03,558 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:03,558 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:55:03,558 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:03,558 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:55:03,558 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:03,567 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/7553b1bc471c43fd97f2193bc8a4bfd5 2023-04-21 01:55:03,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:03,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042163577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,608 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:03,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:03,610 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:03,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:03,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:03,611 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:03,611 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:03,611 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:03,611 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:03,611 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,611 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:03,612 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,612 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:03,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:03,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:03,613 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,613 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:03,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:03,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:03,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:03,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:03,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:03,615 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:03,615 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:03,615 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:03,615 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:03,615 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:03,615 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:03,615 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:03,615 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:03,615 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:03,615 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:03,615 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:03,615 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,615 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:03,615 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:03,621 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,621 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:03,621 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,621 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:03,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:03,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:03,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:03,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63b8011a[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:03,633 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:03,633 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,634 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,634 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:03,634 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:03,634 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:03,634 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,634 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:03,635 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,635 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:03,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:03,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:03,635 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:03,635 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:03,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:03,637 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:03,638 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:03,638 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:03,638 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:03,638 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:03,639 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,639 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,639 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:03,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:03,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:03,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:03,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:03,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:03,640 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:03,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,641 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:03,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,648 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,648 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:03,648 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:03,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:03,648 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:03,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:03,648 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:03,649 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:03,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:03,648 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:03,649 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,649 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,648 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:03,649 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:03,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:03,649 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:03,649 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,649 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:03,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:03,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042163680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,749 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:03,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:03,751 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:03,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:03,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:03,753 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:03,753 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:03,753 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:03,753 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:03,753 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,753 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:03,754 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,754 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:03,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:03,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:03,755 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,755 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:03,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:03,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:03,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:03,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:03,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:03,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:03,756 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:03,756 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:03,756 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:03,756 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:03,756 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:03,756 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:03,756 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:03,756 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:03,756 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:03,757 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:03,757 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:03,757 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,757 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:03,757 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:03,758 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,758 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:03,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:03,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:03,759 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,759 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:03,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:03,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:03,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:03,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5161f298[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:03,760 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:03,760 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,761 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,761 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,761 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:03,761 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:03,761 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:03,761 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:03,763 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,763 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:03,763 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:03,763 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:03,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:03,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:03,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:03,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:03,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:03,765 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:03,765 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:03,765 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:03,765 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:03,765 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:03,767 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,767 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,767 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:03,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:03,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:03,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:03,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:03,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:03,768 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:03,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,769 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:03,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,771 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:03,771 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:03,771 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:03,771 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:03,771 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:03,771 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,771 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:03,772 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:03,771 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:03,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:03,772 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,772 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,772 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:03,772 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:03,772 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:03,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:03,872 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:03,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:03,874 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:03,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:03,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:03,876 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:03,876 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:03,876 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:03,876 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:03,876 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,876 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:03,878 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:03,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:03,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:03,878 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:03,878 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,878 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:03,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:03,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:03,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:03,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:03,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:03,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:03,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:03,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:03,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:03,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:03,880 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:03,880 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:03,880 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:03,880 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:03,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:03,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:03,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:03,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:03,882 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,882 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:03,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:03,883 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,883 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:03,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:03,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:03,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:03,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:03,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38e15756[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:03,885 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:03,885 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:03,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042163884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,886 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,886 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:03,886 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:03,886 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,886 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:03,886 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:03,887 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:03,887 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,887 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:03,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:03,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:03,888 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:03,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:03,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:03,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:03,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:03,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:03,890 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:03,890 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:03,890 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:03,890 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:03,890 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:03,891 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,891 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,891 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:03,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:03,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:03,892 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:03,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:03,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:03,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:03,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,893 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:03,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:03,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,896 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,896 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:03,896 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:03,896 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:03,896 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:03,897 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:03,897 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:03,897 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:03,897 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,897 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:03,897 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:03,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:03,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:03,897 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:03,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:03,897 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:03,897 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:03,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:03,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:03,982 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/dbfa142a52974e65997c72bd7b750a46 2023-04-21 01:55:03,991 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/765ab96305ca4e5b80d030638c56b0da 2023-04-21 01:55:03,995 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/7553b1bc471c43fd97f2193bc8a4bfd5 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/7553b1bc471c43fd97f2193bc8a4bfd5 2023-04-21 01:55:03,997 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:03,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:03,998 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/7553b1bc471c43fd97f2193bc8a4bfd5, entries=200, sequenceid=306, filesize=14.3 K 2023-04-21 01:55:03,999 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/dbfa142a52974e65997c72bd7b750a46 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/dbfa142a52974e65997c72bd7b750a46 2023-04-21 01:55:04,000 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:04,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:04,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:04,001 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:04,001 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:04,001 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:04,001 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:04,005 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/dbfa142a52974e65997c72bd7b750a46, entries=150, sequenceid=306, filesize=11.9 K 2023-04-21 01:55:04,006 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:04,006 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:04,006 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/765ab96305ca4e5b80d030638c56b0da as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/765ab96305ca4e5b80d030638c56b0da 2023-04-21 01:55:04,007 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,007 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:04,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:04,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:04,007 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,007 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:04,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:04,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:04,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:04,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:04,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:04,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:04,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:04,008 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:04,008 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:04,008 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:04,010 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:04,010 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:04,010 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:04,010 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:04,010 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:04,010 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:04,010 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:04,010 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:04,010 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,012 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:04,012 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:04,013 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,013 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:04,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:04,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:04,014 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/765ab96305ca4e5b80d030638c56b0da, entries=150, sequenceid=306, filesize=11.9 K 2023-04-21 01:55:04,014 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:04,014 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:04,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:04,014 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 65bad43ef0e690f33e72a230ffd7ee4f in 457ms, sequenceid=306, compaction requested=true 2023-04-21 01:55:04,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:04,014 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:04,015 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:04,015 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:04,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:04,015 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:04,015 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:04,015 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:04,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:04,016 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36848 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:04,016 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34408 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:04,016 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/A is initiating minor compaction (all files) 2023-04-21 01:55:04,016 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/B is initiating minor compaction (all files) 2023-04-21 01:55:04,016 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/A in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:04,016 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/8aee80d3205b48dfa766bf162416878a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/008f70626533407db9586fb727ed1e24, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/7553b1bc471c43fd97f2193bc8a4bfd5] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=36.0 K 2023-04-21 01:55:04,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:04,016 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/B in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:04,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@117577f6[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:04,016 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/51cda5dcdedf4ef7ae3b5f9d2b6f4145, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/1a58373ef20741a7b40681733ec19f2c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/dbfa142a52974e65997c72bd7b750a46] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=33.6 K 2023-04-21 01:55:04,016 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:04,016 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:04,017 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 8aee80d3205b48dfa766bf162416878a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682042100562 2023-04-21 01:55:04,017 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 51cda5dcdedf4ef7ae3b5f9d2b6f4145, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682042100562 2023-04-21 01:55:04,017 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 008f70626533407db9586fb727ed1e24, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1682042101717 2023-04-21 01:55:04,017 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 1a58373ef20741a7b40681733ec19f2c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1682042101717 2023-04-21 01:55:04,018 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:04,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:04,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:04,018 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:04,018 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:04,018 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:04,018 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 7553b1bc471c43fd97f2193bc8a4bfd5, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1682042102926 2023-04-21 01:55:04,018 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting dbfa142a52974e65997c72bd7b750a46, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1682042102939 2023-04-21 01:55:04,023 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:04,023 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,023 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:04,023 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:04,023 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:04,023 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:04,023 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:04,023 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:04,023 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:04,023 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:04,023 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:04,026 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:04,026 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:04,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:04,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:04,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:04,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:04,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:04,026 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#A#compaction#282 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:04,026 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:04,026 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#B#compaction#283 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:04,027 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:04,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:04,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:04,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:04,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:04,028 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:04,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:04,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:04,031 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:04,031 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:04,031 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:04,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:04,031 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:04,031 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:04,031 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:04,031 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:04,031 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:04,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:04,031 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:04,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:04,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:04,032 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,031 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:04,032 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:04,032 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:04,032 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:04,132 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:04,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:04,134 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:04,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:04,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:04,135 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:04,135 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:04,135 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:04,135 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:04,136 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:04,136 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:04,137 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:04,137 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:04,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:04,138 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,138 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:04,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:04,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:04,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:04,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:04,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:04,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:04,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:04,139 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:04,139 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:04,139 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:04,139 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:04,139 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:04,139 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:04,139 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:04,139 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-21 01:55:04,140 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:55:04,140 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:04,140 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:55:04,140 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:04,140 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:55:04,140 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:04,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:55:04,187 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:04,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042104223, max timestamp=1682042104237Num uniques 859; 2023-04-21 01:55:04,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042104223, max timestamp=1682042104237Num uniques 859; 2023-04-21 01:55:04,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:04,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042104223, max timestamp=1682042104237Num uniques 859; 2023-04-21 01:55:04,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:04,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:04,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:04,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042164287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:04,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042164390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,441 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/627e2933fb2f46598ba1254c164e1f2a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/627e2933fb2f46598ba1254c164e1f2a 2023-04-21 01:55:04,443 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/cd489df9b6954ac7aaaf695ab36c76ce as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/cd489df9b6954ac7aaaf695ab36c76ce 2023-04-21 01:55:04,446 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/B of 65bad43ef0e690f33e72a230ffd7ee4f into 627e2933fb2f46598ba1254c164e1f2a(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:04,446 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:04,446 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/B, priority=13, startTime=1682042104015; duration=0sec 2023-04-21 01:55:04,447 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:04,447 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:04,447 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/A of 65bad43ef0e690f33e72a230ffd7ee4f into cd489df9b6954ac7aaaf695ab36c76ce(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:04,447 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:04,447 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/A, priority=13, startTime=1682042104015; duration=0sec 2023-04-21 01:55:04,448 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:04,448 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34408 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:04,448 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/C is initiating minor compaction (all files) 2023-04-21 01:55:04,448 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/C in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:04,448 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/6147ebe5828e418499e07714cdfd153c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a3094ddd28854e1897129b10ac8628b0, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/765ab96305ca4e5b80d030638c56b0da] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=33.6 K 2023-04-21 01:55:04,448 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 6147ebe5828e418499e07714cdfd153c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682042100562 2023-04-21 01:55:04,449 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting a3094ddd28854e1897129b10ac8628b0, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1682042101717 2023-04-21 01:55:04,449 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 765ab96305ca4e5b80d030638c56b0da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1682042102939 2023-04-21 01:55:04,454 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#C#compaction#285 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:04,461 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/56d718cf290a47d4940763286c27fff5 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/56d718cf290a47d4940763286c27fff5 2023-04-21 01:55:04,465 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/C of 65bad43ef0e690f33e72a230ffd7ee4f into 56d718cf290a47d4940763286c27fff5(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:04,465 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:04,465 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/C, priority=13, startTime=1682042104015; duration=0sec 2023-04-21 01:55:04,465 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:04,548 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/864b6ef7ea6b4dc680db9c1c578317a3 2023-04-21 01:55:04,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:04,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042164594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:04,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042164899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,959 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/f29a5509c228496d8190c708adfa28d3 2023-04-21 01:55:04,969 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/a2e04bbdd9ae47f8a156ed67705eaa55 2023-04-21 01:55:04,973 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/864b6ef7ea6b4dc680db9c1c578317a3 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/864b6ef7ea6b4dc680db9c1c578317a3 2023-04-21 01:55:04,976 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/864b6ef7ea6b4dc680db9c1c578317a3, entries=150, sequenceid=316, filesize=11.9 K 2023-04-21 01:55:04,977 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/f29a5509c228496d8190c708adfa28d3 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/f29a5509c228496d8190c708adfa28d3 2023-04-21 01:55:04,981 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/f29a5509c228496d8190c708adfa28d3, entries=150, sequenceid=316, filesize=11.9 K 2023-04-21 01:55:04,981 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/a2e04bbdd9ae47f8a156ed67705eaa55 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a2e04bbdd9ae47f8a156ed67705eaa55 2023-04-21 01:55:04,984 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a2e04bbdd9ae47f8a156ed67705eaa55, entries=150, sequenceid=316, filesize=11.9 K 2023-04-21 01:55:04,985 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 65bad43ef0e690f33e72a230ffd7ee4f in 846ms, sequenceid=316, compaction requested=false 2023-04-21 01:55:04,985 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:04,985 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:04,985 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:04,985 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:04,985 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,985 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:04,985 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:04,987 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,987 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:04,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:04,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:04,987 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:04,988 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:04,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:04,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:04,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:04,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:04,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:04,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32e60413[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:04,989 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:04,989 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:04,990 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:04,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:04,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:04,991 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:04,991 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:04,991 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:04,991 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,991 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:04,992 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:04,992 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,992 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:04,992 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:04,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:04,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:04,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:04,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:04,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:04,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:04,995 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:04,995 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:04,995 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:04,995 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:04,995 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:04,998 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:04,998 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:04,998 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:04,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:04,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:04,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:04,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:04,999 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:04,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:05,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:05,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:05,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,001 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:05,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:05,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,004 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,004 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:05,004 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:05,004 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:05,004 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:05,004 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:05,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:05,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:05,005 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:05,005 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:05,004 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:05,004 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:05,005 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:05,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:05,005 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:05,005 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:05,005 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:05,105 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:05,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:05,107 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:05,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:05,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:05,108 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:05,108 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:05,108 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:05,108 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:05,108 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:05,109 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:05,110 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,110 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:05,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:05,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:05,110 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,110 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:05,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:05,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:05,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:05,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:05,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:05,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:05,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:05,111 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:05,111 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:05,111 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:05,112 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:05,112 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:05,112 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:05,112 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:05,112 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-21 01:55:05,112 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:55:05,112 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:05,112 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:55:05,112 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:05,112 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:55:05,112 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:05,123 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/7781c36b19404e8cb991688f9004ced4 2023-04-21 01:55:05,133 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/0bf4217ea945471f9e3d48f7e9c9cc67 2023-04-21 01:55:05,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:55:05,407 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:05,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:05,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042165427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:05,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042165531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,544 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/8950d96fba414308893cee9524fd7fd6 2023-04-21 01:55:05,548 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/7781c36b19404e8cb991688f9004ced4 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/7781c36b19404e8cb991688f9004ced4 2023-04-21 01:55:05,551 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/7781c36b19404e8cb991688f9004ced4, entries=150, sequenceid=356, filesize=11.9 K 2023-04-21 01:55:05,552 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/0bf4217ea945471f9e3d48f7e9c9cc67 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/0bf4217ea945471f9e3d48f7e9c9cc67 2023-04-21 01:55:05,557 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/0bf4217ea945471f9e3d48f7e9c9cc67, entries=150, sequenceid=356, filesize=11.9 K 2023-04-21 01:55:05,557 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/8950d96fba414308893cee9524fd7fd6 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/8950d96fba414308893cee9524fd7fd6 2023-04-21 01:55:05,560 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/8950d96fba414308893cee9524fd7fd6, entries=150, sequenceid=356, filesize=11.9 K 2023-04-21 01:55:05,561 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 65bad43ef0e690f33e72a230ffd7ee4f in 449ms, sequenceid=356, compaction requested=true 2023-04-21 01:55:05,561 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:05,561 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:05,561 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:05,561 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:05,561 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,561 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:05,561 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:05,563 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:05,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:05,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:05,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:05,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:05,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:05,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:05,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c4dd5e5[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:05,565 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:05,565 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:05,567 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:05,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:05,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:05,567 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:05,567 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:05,567 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:05,567 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,567 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:05,568 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,568 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:05,568 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:05,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:05,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:05,568 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:05,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:05,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:05,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:05,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:05,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:05,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:05,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:05,571 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:05,571 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:05,571 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:05,571 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:05,571 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:05,573 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:05,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:05,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:05,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:05,573 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:05,573 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:05,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:05,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:05,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:05,573 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:05,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:05,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:05,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:05,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:05,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:05,574 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:05,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:05,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,577 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:05,577 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:05,577 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,577 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:05,577 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:05,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:05,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:05,577 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:05,577 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:05,577 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:05,577 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:05,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:05,577 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:05,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:05,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:05,578 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,578 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:05,578 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:05,578 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:05,677 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:05,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:05,680 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:05,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:05,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:05,681 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:05,681 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:05,681 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:05,681 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:05,681 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:05,681 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:05,684 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:05,684 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:05,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:05,684 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:05,684 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:05,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:05,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:05,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:05,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:05,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:05,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:05,685 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:05,685 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:05,685 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:05,685 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:05,685 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:05,686 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:05,686 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:05,686 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-21 01:55:05,686 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:55:05,686 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:05,686 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:55:05,686 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:05,686 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:55:05,686 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:05,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:55:05,737 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:05,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042105765, max timestamp=1682042105782Num uniques 859; 2023-04-21 01:55:05,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:05,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042105765, max timestamp=1682042105782Num uniques 859; 2023-04-21 01:55:05,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042105765, max timestamp=1682042105782Num uniques 859; 2023-04-21 01:55:05,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:05,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:05,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:05,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042165842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:05,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042165948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:05,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52724 deadline: 1682042165979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,983 DEBUG [Thread-1860] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:55:05,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:05,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52788 deadline: 1682042165991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,994 DEBUG [Thread-1864] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:55:05,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:05,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52738 deadline: 1682042165991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:05,996 DEBUG [Thread-1866] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8173 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:55:06,094 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/7a0392f7d93a435b84bb4d752ae61545 2023-04-21 01:55:06,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:06,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042166151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:06,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:06,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042166454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:06,505 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/1aec52510c4f42fdb9c6818916233268 2023-04-21 01:55:06,916 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/a78803a333ea47b58535beb30011ce2d 2023-04-21 01:55:06,920 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/7a0392f7d93a435b84bb4d752ae61545 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/7a0392f7d93a435b84bb4d752ae61545 2023-04-21 01:55:06,923 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/7a0392f7d93a435b84bb4d752ae61545, entries=150, sequenceid=366, filesize=11.9 K 2023-04-21 01:55:06,924 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/1aec52510c4f42fdb9c6818916233268 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/1aec52510c4f42fdb9c6818916233268 2023-04-21 01:55:06,928 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/1aec52510c4f42fdb9c6818916233268, entries=150, sequenceid=366, filesize=11.9 K 2023-04-21 01:55:06,928 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/a78803a333ea47b58535beb30011ce2d as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a78803a333ea47b58535beb30011ce2d 2023-04-21 01:55:06,933 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a78803a333ea47b58535beb30011ce2d, entries=150, sequenceid=366, filesize=11.9 K 2023-04-21 01:55:06,934 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 65bad43ef0e690f33e72a230ffd7ee4f in 1248ms, sequenceid=366, compaction requested=true 2023-04-21 01:55:06,934 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:06,934 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:06,934 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:06,934 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:06,934 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:06,934 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:06,934 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:06,936 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:06,936 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:06,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:06,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:06,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:06,936 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:06,936 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:06,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:06,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:06,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:06,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:06,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:06,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:06,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ea474dd[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:06,938 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:06,938 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:06,939 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:06,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:06,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:06,939 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:06,939 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:06,939 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:06,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:06,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:06,940 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:06,940 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:06,941 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:06,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:06,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:06,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:06,941 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:06,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:06,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:06,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:06,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:06,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:06,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:06,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:06,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:06,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:06,943 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:06,943 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:06,943 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:06,943 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:06,943 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:06,948 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:06,948 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:06,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:06,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:06,948 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:06,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:06,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:06,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:06,948 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:06,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:06,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:06,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:06,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:06,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:06,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:06,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:06,950 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:06,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:06,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:06,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:06,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:06,952 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:06,952 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:06,952 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:06,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:06,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:06,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:06,952 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:06,952 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:06,952 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:06,952 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:06,952 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:06,952 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:06,952 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:06,952 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:06,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:06,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:06,952 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:06,953 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:06,953 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:06,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:55:06,963 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-21 01:55:06,964 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:55:06,964 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-21 01:55:06,964 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:55:06,964 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-21 01:55:06,964 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:55:06,964 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-21 01:55:06,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:06,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042166991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,053 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:07,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:07,055 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:07,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:07,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:07,057 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:07,057 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:07,057 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:07,057 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:07,057 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,057 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,058 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,058 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:07,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:07,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:07,059 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,059 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:07,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:07,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:07,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:07,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:07,060 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:07,060 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:07,060 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:07,060 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:07,060 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:07,060 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:07,060 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:07,060 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:07,060 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:07,060 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:07,061 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:07,061 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,061 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:07,061 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:07,062 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,062 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,062 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,062 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:07,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:07,064 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:07,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64547238[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:07,064 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,065 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,066 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:07,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:07,066 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:07,066 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:07,067 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:07,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:07,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:07,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:07,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:07,069 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:07,069 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:07,069 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:07,069 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:07,069 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:07,071 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,071 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,071 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:07,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,071 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:07,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:07,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,072 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,075 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,075 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:07,075 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:07,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:07,075 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:07,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:07,075 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:07,075 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:07,075 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:07,075 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:07,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:07,076 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,076 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,076 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,076 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:07,076 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:07,076 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:07,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042167095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,176 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:07,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:07,178 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:07,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:07,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:07,180 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:07,180 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:07,180 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:07,180 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:07,180 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,180 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,188 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,188 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:07,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:07,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:07,188 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,188 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:07,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:07,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:07,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:07,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:07,189 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:07,189 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:07,189 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:07,189 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:07,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:07,190 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:07,190 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:07,190 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:07,190 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:07,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:07,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:07,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:07,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:07,191 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,191 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:07,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:07,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6610243d[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:07,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:07,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,194 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,194 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:07,194 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:07,194 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,194 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:07,194 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:07,196 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,196 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:07,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,196 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:07,196 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:07,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:07,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:07,198 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:07,198 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:07,198 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:07,198 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:07,198 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:07,205 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,205 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,205 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:07,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:07,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:07,206 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,207 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,209 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,209 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,209 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:07,209 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:07,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:07,209 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:07,209 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:07,209 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,209 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:07,209 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:07,209 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:07,210 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:07,210 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:07,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:07,210 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,210 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:07,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:07,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042167298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,310 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:07,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:07,312 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:07,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:07,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:07,313 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:07,313 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:07,314 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:07,314 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:07,314 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,314 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,315 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,315 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:07,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:07,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:07,315 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,315 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:07,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:07,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:07,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:07,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:07,317 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:07,317 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:07,317 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:07,317 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:07,317 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:07,317 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:07,317 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:07,317 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:07,317 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:07,317 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:07,317 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:07,317 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,317 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:07,317 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:07,319 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,319 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,319 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,319 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:07,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:07,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@647027e7[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:07,320 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:07,320 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,322 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,322 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:07,322 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:07,322 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:07,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:07,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:07,323 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:07,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:07,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:07,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:07,326 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:07,326 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:07,326 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:07,326 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:07,326 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:07,328 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,328 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,328 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:07,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:07,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:07,328 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,329 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,336 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,336 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,336 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,336 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,336 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:07,336 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:07,336 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:07,336 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:07,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:07,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:07,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:07,336 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:07,336 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,336 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:07,336 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,336 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:07,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:07,336 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:07,373 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/9c6034905a364c2093c670490aab92f7 2023-04-21 01:55:07,436 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:07,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:07,439 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:07,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:07,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:07,440 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:07,440 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:07,440 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:07,440 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:07,440 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,440 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,442 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,442 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:07,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:07,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:07,442 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,442 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:07,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:07,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:07,443 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:07,443 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:07,444 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:07,444 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:07,444 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:07,444 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:07,444 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:07,444 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:07,444 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:07,444 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:07,444 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:07,444 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:07,444 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:07,444 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,444 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:07,444 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:07,446 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,446 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,446 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,446 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:07,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:07,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@190fb528[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:07,448 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:07,448 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,449 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,449 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:07,449 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:07,449 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:07,449 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,450 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:07,451 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,451 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:07,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,451 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:07,451 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:07,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:07,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:07,454 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:07,454 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:07,454 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:07,454 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:07,454 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:07,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:07,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:07,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:07,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,457 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,458 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,460 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:07,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:07,460 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:07,460 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:07,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:07,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:07,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:07,460 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:07,460 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:07,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:07,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:07,560 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:07,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:07,562 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:07,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:07,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:07,563 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:07,564 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:07,564 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:07,564 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:07,564 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,564 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,565 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,565 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:07,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:07,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:07,566 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,566 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:07,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:07,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,567 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:07,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:07,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:07,567 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:07,567 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:07,567 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:07,567 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:07,567 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:07,567 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:07,567 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:07,567 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:07,567 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:07,568 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:07,568 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:07,568 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,568 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:07,568 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:07,570 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,570 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,570 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,570 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:07,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:07,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e0dc183[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:07,572 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:07,572 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,573 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,573 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:07,573 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:07,573 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:07,573 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,573 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:07,574 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:07,574 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,574 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:07,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,575 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:07,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:07,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:07,577 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:07,577 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:07,577 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:07,577 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:07,577 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:07,579 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,579 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:07,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,579 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:07,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:07,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,579 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,580 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,583 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,583 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,583 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,583 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:07,584 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:07,583 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:07,584 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:07,584 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:07,584 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:07,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:07,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:07,584 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,584 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,584 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:07,584 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:07,584 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:07,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:07,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:07,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042167602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,684 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:07,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:07,686 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:07,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:07,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:07,688 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:07,688 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:07,688 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:07,688 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:07,688 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,688 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,689 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,689 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:07,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:07,690 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,690 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:07,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:07,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:07,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:07,691 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:07,691 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:07,691 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:07,691 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:07,691 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:07,691 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:07,692 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:07,692 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:07,692 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:07,692 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:07,692 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:07,692 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,692 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:07,692 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:07,693 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,693 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:07,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:07,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@395b27b3[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:07,695 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:07,695 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,696 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,697 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:07,697 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:07,697 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:07,697 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,697 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:07,699 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,699 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,699 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:07,699 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:07,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:07,702 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:07,702 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:07,702 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:07,702 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:07,702 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:07,703 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,703 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:07,703 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:07,704 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:07,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,705 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,708 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:07,708 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,708 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:07,708 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:07,709 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:07,708 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:07,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:07,709 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:07,709 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:07,708 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:07,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:07,709 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,709 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:07,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:07,709 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,709 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,709 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,783 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/8d57cea147df416abf76ab84e5e7a8e5 2023-04-21 01:55:07,809 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:07,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:07,811 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:07,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:07,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:07,812 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:07,812 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:07,812 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:07,812 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:07,813 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,813 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,817 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:07,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:07,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:07,818 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,818 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:07,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:07,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:07,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:07,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:07,819 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:07,819 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:07,819 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:07,819 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:07,819 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:07,819 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:07,819 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:07,819 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:07,819 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:07,819 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:07,819 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:07,819 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,819 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:07,819 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:07,821 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,821 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,821 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,821 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:07,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:07,822 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:07,822 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cf16060[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:07,823 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,824 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:07,824 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:07,824 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:07,824 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,824 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:07,825 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,825 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:07,825 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,825 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:07,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:07,828 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:07,828 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:07,828 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:07,828 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:07,828 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:07,830 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,830 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,830 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:07,831 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,832 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:07,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:07,834 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:07,834 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:07,834 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:07,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:07,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:07,835 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:07,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,835 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:07,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:07,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:07,935 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:07,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:07,937 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:07,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:07,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:07,938 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:07,938 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:07,938 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:07,938 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:07,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,940 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,940 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:07,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:07,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:07,940 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,940 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:07,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:07,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:07,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:07,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:07,941 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:07,941 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:07,941 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:07,942 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:07,942 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:07,942 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:07,942 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:07,942 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:07,942 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:07,942 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:07,942 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:07,942 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,942 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:07,942 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:07,943 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,943 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,944 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,944 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:07,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:07,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38333823[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:07,945 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:07,945 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,948 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,948 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,948 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:07,948 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:07,948 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:07,948 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:07,949 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,949 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,949 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:07,950 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:07,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:07,956 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:07,956 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:07,956 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:07,956 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:07,956 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:07,957 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,957 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:07,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,958 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:07,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:07,958 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:07,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,959 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:07,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:07,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,962 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,962 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,962 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:07,962 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:07,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:07,962 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:07,962 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:07,962 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:07,962 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:07,962 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:07,962 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,962 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:07,962 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:07,962 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:07,962 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:07,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:07,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:07,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:08,063 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:08,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:08,065 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:08,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:08,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:08,066 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:08,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:08,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:08,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:08,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:08,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:08,068 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:08,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:08,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:08,068 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:08,069 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,069 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:08,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:08,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:08,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:08,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:08,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:08,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:08,070 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:08,070 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:08,070 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:08,070 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:08,070 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:08,070 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:08,070 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:08,070 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:08,070 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:08,070 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:08,070 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:08,070 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,070 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:08,070 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:08,072 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:08,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:08,072 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:08,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:08,072 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:08,072 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:08,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:08,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:08,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:08,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:08,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@882e77f[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:08,073 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:08,073 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:08,074 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:08,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:08,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:08,075 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:08,075 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:08,075 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:08,075 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,075 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:08,076 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,076 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:08,076 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:08,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:08,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:08,076 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:08,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:08,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:08,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:08,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:08,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:08,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:08,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:08,078 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:08,078 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:08,078 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:08,078 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:08,078 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:08,081 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:08,081 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:08,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:08,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:08,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:08,081 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:08,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:08,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:08,082 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:08,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:08,083 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:08,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:08,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:08,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,085 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,085 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:08,085 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:08,085 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:08,085 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:08,085 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:08,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:08,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:08,085 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:08,085 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:08,085 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:08,085 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:08,085 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,085 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:08,085 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:08,085 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:08,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:08,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:08,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:08,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:08,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042168108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,185 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:08,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:08,187 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:08,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:08,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:08,189 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:08,189 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:08,189 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:08,189 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:08,189 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:08,189 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:08,190 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,190 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:08,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:08,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:08,191 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,191 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:08,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:08,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:08,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:08,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:08,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:08,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:08,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:08,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:08,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:08,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:08,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:08,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:08,192 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:08,192 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:08,192 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:08,192 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:08,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:08,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:08,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:08,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:08,193 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/4393802e223345588b1df85bc8215499 2023-04-21 01:55:08,199 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,199 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:08,199 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:08,199 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:08,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:08,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:08,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:08,201 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/9c6034905a364c2093c670490aab92f7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/9c6034905a364c2093c670490aab92f7 2023-04-21 01:55:08,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:08,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68842f63[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:08,202 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:08,202 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:08,203 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:08,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:08,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:08,203 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:08,203 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:08,203 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:08,203 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,203 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:08,204 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,204 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:08,204 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:08,204 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:08,205 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/9c6034905a364c2093c670490aab92f7, entries=200, sequenceid=404, filesize=14.3 K 2023-04-21 01:55:08,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:08,206 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/8d57cea147df416abf76ab84e5e7a8e5 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/8d57cea147df416abf76ab84e5e7a8e5 2023-04-21 01:55:08,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:08,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:08,208 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:08,208 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:08,208 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:08,208 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:08,208 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:08,210 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:08,210 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:08,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:08,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:08,211 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/8d57cea147df416abf76ab84e5e7a8e5, entries=150, sequenceid=404, filesize=11.9 K 2023-04-21 01:55:08,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:08,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:08,211 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:08,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:08,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:08,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:08,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:08,212 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/4393802e223345588b1df85bc8215499 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/4393802e223345588b1df85bc8215499 2023-04-21 01:55:08,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:08,212 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:08,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:08,213 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:08,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:08,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,215 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/4393802e223345588b1df85bc8215499, entries=150, sequenceid=404, filesize=11.9 K 2023-04-21 01:55:08,216 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 65bad43ef0e690f33e72a230ffd7ee4f in 1253ms, sequenceid=404, compaction requested=true 2023-04-21 01:55:08,216 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:08,216 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:08,216 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-21 01:55:08,216 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:08,216 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-21 01:55:08,217 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:08,217 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,217 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:08,217 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:08,217 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:08,217 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:08,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:08,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:08,217 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:08,217 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:08,217 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:08,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:08,217 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-21 01:55:08,217 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:08,217 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,217 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:08,217 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:08,217 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:08,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:08,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:08,218 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 63916 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-21 01:55:08,218 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/A is initiating minor compaction (all files) 2023-04-21 01:55:08,219 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/A in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:08,219 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/cd489df9b6954ac7aaaf695ab36c76ce, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/864b6ef7ea6b4dc680db9c1c578317a3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/7781c36b19404e8cb991688f9004ced4, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/7a0392f7d93a435b84bb4d752ae61545, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/9c6034905a364c2093c670490aab92f7] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=62.4 K 2023-04-21 01:55:08,219 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting cd489df9b6954ac7aaaf695ab36c76ce, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1682042102939 2023-04-21 01:55:08,219 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61476 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-21 01:55:08,219 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/B is initiating minor compaction (all files) 2023-04-21 01:55:08,219 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/B in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:08,219 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 864b6ef7ea6b4dc680db9c1c578317a3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1682042103567 2023-04-21 01:55:08,219 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/627e2933fb2f46598ba1254c164e1f2a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/f29a5509c228496d8190c708adfa28d3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/0bf4217ea945471f9e3d48f7e9c9cc67, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/1aec52510c4f42fdb9c6818916233268, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/8d57cea147df416abf76ab84e5e7a8e5] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=60.0 K 2023-04-21 01:55:08,220 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 7781c36b19404e8cb991688f9004ced4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1682042104257 2023-04-21 01:55:08,220 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 627e2933fb2f46598ba1254c164e1f2a, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1682042102939 2023-04-21 01:55:08,221 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 7a0392f7d93a435b84bb4d752ae61545, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682042105410 2023-04-21 01:55:08,221 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting f29a5509c228496d8190c708adfa28d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1682042103567 2023-04-21 01:55:08,221 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 9c6034905a364c2093c670490aab92f7, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1682042105835 2023-04-21 01:55:08,221 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 0bf4217ea945471f9e3d48f7e9c9cc67, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1682042104257 2023-04-21 01:55:08,222 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 1aec52510c4f42fdb9c6818916233268, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682042105410 2023-04-21 01:55:08,222 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 8d57cea147df416abf76ab84e5e7a8e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1682042105838 2023-04-21 01:55:08,230 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#A#compaction#297 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:08,234 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#B#compaction#298 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:08,317 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:08,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:08,319 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:08,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:08,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:08,320 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:08,320 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:08,320 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:08,320 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:08,321 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:08,321 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:08,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,322 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:08,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:08,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:08,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:08,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:08,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:08,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:08,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:08,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:08,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:08,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:08,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:08,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:08,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:08,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:08,324 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:08,324 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:08,324 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:08,324 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:08,324 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-21 01:55:08,324 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:55:08,324 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:08,324 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:55:08,324 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:08,324 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:55:08,324 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:08,642 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/894ec71e5bad4ed3ae0136e2eccff7a4 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/894ec71e5bad4ed3ae0136e2eccff7a4 2023-04-21 01:55:08,648 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/d0584eb9365a42faa0b974d8b46fd174 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/d0584eb9365a42faa0b974d8b46fd174 2023-04-21 01:55:08,660 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/A of 65bad43ef0e690f33e72a230ffd7ee4f into 894ec71e5bad4ed3ae0136e2eccff7a4(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:08,660 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:08,660 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/A, priority=11, startTime=1682042108216; duration=0sec 2023-04-21 01:55:08,660 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:08,661 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-21 01:55:08,663 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61476 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-21 01:55:08,663 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/C is initiating minor compaction (all files) 2023-04-21 01:55:08,663 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/C in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:08,663 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/56d718cf290a47d4940763286c27fff5, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a2e04bbdd9ae47f8a156ed67705eaa55, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/8950d96fba414308893cee9524fd7fd6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a78803a333ea47b58535beb30011ce2d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/4393802e223345588b1df85bc8215499] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=60.0 K 2023-04-21 01:55:08,663 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 56d718cf290a47d4940763286c27fff5, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1682042102939 2023-04-21 01:55:08,664 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting a2e04bbdd9ae47f8a156ed67705eaa55, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1682042103567 2023-04-21 01:55:08,665 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 8950d96fba414308893cee9524fd7fd6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1682042104257 2023-04-21 01:55:08,665 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting a78803a333ea47b58535beb30011ce2d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682042105410 2023-04-21 01:55:08,665 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 4393802e223345588b1df85bc8215499, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1682042105838 2023-04-21 01:55:08,674 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/B of 65bad43ef0e690f33e72a230ffd7ee4f into d0584eb9365a42faa0b974d8b46fd174(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:08,674 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:08,674 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/B, priority=11, startTime=1682042108216; duration=0sec 2023-04-21 01:55:08,674 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:08,678 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#C#compaction#300 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:08,689 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/2615a761556840eba764eedd8ab8ffcc as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/2615a761556840eba764eedd8ab8ffcc 2023-04-21 01:55:08,694 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/C of 65bad43ef0e690f33e72a230ffd7ee4f into 2615a761556840eba764eedd8ab8ffcc(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:08,694 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:08,694 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/C, priority=11, startTime=1682042108216; duration=0sec 2023-04-21 01:55:08,694 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:08,731 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/2ffb55d109d1429e88c57de62c3d5d85 2023-04-21 01:55:08,755 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/051f571ec38249d385386b72770f33a0 2023-04-21 01:55:09,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:55:09,113 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:09,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042109137, max timestamp=1682042109151Num uniques 859; 2023-04-21 01:55:09,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042109137, max timestamp=1682042109151Num uniques 859; 2023-04-21 01:55:09,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042109137, max timestamp=1682042109151Num uniques 859; 2023-04-21 01:55:09,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:09,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:09,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:09,170 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/ebb0c3c3c03246d2a731d09d09f0515e 2023-04-21 01:55:09,174 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/2ffb55d109d1429e88c57de62c3d5d85 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/2ffb55d109d1429e88c57de62c3d5d85 2023-04-21 01:55:09,178 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/2ffb55d109d1429e88c57de62c3d5d85, entries=150, sequenceid=413, filesize=11.9 K 2023-04-21 01:55:09,178 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/051f571ec38249d385386b72770f33a0 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/051f571ec38249d385386b72770f33a0 2023-04-21 01:55:09,182 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/051f571ec38249d385386b72770f33a0, entries=150, sequenceid=413, filesize=11.9 K 2023-04-21 01:55:09,182 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/ebb0c3c3c03246d2a731d09d09f0515e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/ebb0c3c3c03246d2a731d09d09f0515e 2023-04-21 01:55:09,186 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/ebb0c3c3c03246d2a731d09d09f0515e, entries=150, sequenceid=413, filesize=11.9 K 2023-04-21 01:55:09,186 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=93.93 KB/96180 for 65bad43ef0e690f33e72a230ffd7ee4f in 862ms, sequenceid=413, compaction requested=false 2023-04-21 01:55:09,186 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:09,186 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:09,186 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:09,186 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:09,187 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,187 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:09,187 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:09,188 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,188 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:09,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:09,188 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,188 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:09,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:55:09,189 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=100.63 KB heapSize=285.38 KB 2023-04-21 01:55:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:55:09,189 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:09,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:55:09,189 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:55:09,190 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:09,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:09,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:09,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fa1a073[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:09,190 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:09,190 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,192 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:09,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:09,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:09,192 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,192 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:09,196 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,196 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:09,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:09,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:09,196 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:09,196 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:09,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:09,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:09,197 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/bd7b9ce40ca4413aab9bc90f166c6757 2023-04-21 01:55:09,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:09,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:09,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:09,200 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:09,200 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:09,200 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:09,200 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:09,200 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:09,201 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,201 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:09,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:09,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,201 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:09,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:09,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:09,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:09,202 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:09,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,204 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:09,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,206 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:09,206 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:09,206 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,206 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:09,206 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,206 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:09,206 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:09,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:09,206 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:09,207 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:09,206 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:09,207 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,207 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:09,207 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,207 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:09,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:09,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:09,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042169232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,307 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:09,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:09,309 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:09,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:09,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:09,310 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:09,310 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:09,310 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:09,310 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:09,311 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,311 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,312 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:09,312 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:09,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:09,312 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,312 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:09,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:09,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:09,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:09,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:09,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:09,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:09,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:09,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:09,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:09,314 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:09,314 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:09,314 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:09,314 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:09,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:09,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:09,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:09,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:09,316 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,316 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:09,317 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,317 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:09,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:09,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ef27584[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:09,318 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:09,318 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,319 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,320 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:09,320 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:09,320 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,320 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:09,320 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:09,321 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,321 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:09,321 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:09,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:09,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:09,321 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:09,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:09,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:09,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:09,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:09,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:09,324 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:09,324 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:09,324 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:09,324 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:09,324 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:09,325 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,325 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,325 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:09,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:09,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:09,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:09,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:09,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:09,326 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:09,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,328 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:09,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,331 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,331 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:09,331 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,331 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:09,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:09,331 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:09,331 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:09,331 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,331 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,331 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:09,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:09,332 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:09,332 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:09,332 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:09,332 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:09,332 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:09,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:09,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 379 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042169337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,432 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:09,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:09,435 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:09,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:09,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:09,436 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:09,436 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:09,436 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:09,436 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:09,437 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,437 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,438 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,438 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:09,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:09,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:09,438 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,438 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:09,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:09,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:09,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:09,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:09,439 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:09,439 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:09,439 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:09,440 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:09,440 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:09,440 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:09,440 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:09,440 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:09,440 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:09,440 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:09,440 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:09,440 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,440 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:09,440 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:09,441 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,441 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:09,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:09,442 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,442 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:09,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:09,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:09,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:09,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:09,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d4c03a[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:09,443 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:09,443 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,444 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,444 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:09,444 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:09,445 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:09,445 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,445 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:09,449 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,449 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:09,449 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:09,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:09,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:09,449 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:09,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:09,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:09,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:09,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:09,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:09,451 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:09,451 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:09,451 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:09,451 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:09,451 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:09,453 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,453 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:09,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:09,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,453 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:09,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:09,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:09,453 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:09,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:09,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,455 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:09,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,457 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,457 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,457 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,457 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,457 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:09,457 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:09,457 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:09,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:09,457 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:09,457 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:09,457 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,457 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,457 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:09,458 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:09,458 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:09,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:09,542 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:09,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042169540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,558 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:09,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:09,560 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:09,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:09,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:09,561 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:09,561 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:09,561 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:09,561 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:09,561 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,561 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,562 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:09,562 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:09,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:09,563 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,563 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:09,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:09,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:09,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:09,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:09,564 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:09,564 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:09,564 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:09,564 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:09,564 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:09,564 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:09,564 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:09,564 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:09,564 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:09,564 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:09,564 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:09,564 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,564 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:09,564 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:09,571 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,572 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:09,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:09,572 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,572 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:09,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:09,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:09,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:09,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:09,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48e20854[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:09,573 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:09,573 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,574 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,575 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:09,575 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:09,575 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:09,575 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,575 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:09,576 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,576 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:09,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:09,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:09,576 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:09,576 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:09,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:09,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:09,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:09,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:09,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:09,578 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:09,578 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:09,578 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:09,578 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:09,578 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:09,580 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,580 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:09,580 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:09,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:09,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:09,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:09,580 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:09,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:09,581 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:09,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,586 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,586 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,586 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:09,586 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:09,586 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,586 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,586 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:09,586 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:09,586 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:09,587 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:09,586 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,587 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:09,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:09,587 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:09,587 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:09,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:09,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:09,611 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/a64fae16e7194454b0823a76701ba4f0 2023-04-21 01:55:09,687 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:09,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:09,689 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:09,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:09,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:09,690 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:09,690 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:09,690 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:09,690 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:09,691 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,691 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,692 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:09,692 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:09,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:09,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,693 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,693 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:09,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:09,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:09,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:09,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:09,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:09,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:09,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:09,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:09,694 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:09,694 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:09,694 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:09,694 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:09,695 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:09,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:09,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:09,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:09,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:09,697 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,697 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:09,697 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,697 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:09,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:09,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:09,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:09,699 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:09,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a728b03[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:09,699 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,700 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,700 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:09,700 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:09,700 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:09,700 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,700 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:09,702 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,702 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:09,702 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:09,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:09,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:09,702 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:09,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:09,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:09,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:09,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:09,714 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:09,714 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:09,714 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:09,714 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:09,714 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:09,715 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,715 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,715 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:09,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:09,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:09,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:09,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:09,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:09,716 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:09,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,718 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:09,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,720 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,720 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,720 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,720 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:09,720 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:09,720 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:09,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:09,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:09,720 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:09,720 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:09,721 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:09,721 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:09,720 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:09,720 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,721 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,721 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:09,821 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:09,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:09,823 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:09,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:09,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:09,824 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:09,824 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:09,824 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:09,824 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:09,825 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,825 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,826 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:09,826 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:09,826 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,826 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:09,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:09,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:09,827 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:09,827 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:09,827 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:09,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:09,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:09,828 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:09,828 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:09,828 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:09,828 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:09,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:09,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:09,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:09,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:09,830 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,830 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:09,830 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,831 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:09,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:09,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77219f9c[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:09,832 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:09,832 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,833 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,834 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:09,834 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:09,834 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,834 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:09,834 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:09,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,835 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:09,835 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:09,835 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:09,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:09,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:09,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:09,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:09,838 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:09,838 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:09,838 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:09,838 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:09,838 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:09,843 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,843 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:09,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:09,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,843 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:09,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:09,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:09,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:09,843 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:09,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,844 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:09,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,846 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,846 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,846 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,846 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:09,846 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:09,846 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:09,846 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,846 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:09,847 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:09,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:09,847 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,847 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:09,847 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,847 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:09,847 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:09,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:09,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:09,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:09,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042169844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,947 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:09,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:09,949 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:09,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:09,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:09,950 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:09,950 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:09,950 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:09,950 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:09,951 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,951 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,952 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,952 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:09,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:09,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:09,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,953 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,953 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:09,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:09,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:09,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:09,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:09,954 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:09,954 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:09,954 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:09,954 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:09,954 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:09,954 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:09,954 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:09,954 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:09,954 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:09,954 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:09,954 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:09,954 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,954 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:09,954 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:09,956 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,956 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:09,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:09,956 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,956 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:09,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:09,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:09,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:09,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:09,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50534452[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:09,958 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:09,958 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,960 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,960 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:09,960 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:09,960 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:09,960 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,960 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:09,962 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,962 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:09,962 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:09,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:09,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:09,962 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:09,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:09,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:09,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:09,964 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:09,964 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:09,964 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:09,964 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:09,964 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:09,966 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,966 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,966 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:09,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:09,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:09,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:09,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:09,967 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:09,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,968 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:09,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:09,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,970 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,970 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,970 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:09,970 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:09,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:09,970 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:09,970 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:09,970 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:09,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:09,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:09,971 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:09,971 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:09,970 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:09,970 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:09,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:09,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:09,971 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,971 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:09,971 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:10,021 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/c28242ffaa9c4403a66f5856b794d48a 2023-04-21 01:55:10,026 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/bd7b9ce40ca4413aab9bc90f166c6757 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/bd7b9ce40ca4413aab9bc90f166c6757 2023-04-21 01:55:10,029 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/bd7b9ce40ca4413aab9bc90f166c6757, entries=200, sequenceid=448, filesize=14.3 K 2023-04-21 01:55:10,030 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/a64fae16e7194454b0823a76701ba4f0 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/a64fae16e7194454b0823a76701ba4f0 2023-04-21 01:55:10,033 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/a64fae16e7194454b0823a76701ba4f0, entries=150, sequenceid=448, filesize=11.9 K 2023-04-21 01:55:10,034 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/c28242ffaa9c4403a66f5856b794d48a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/c28242ffaa9c4403a66f5856b794d48a 2023-04-21 01:55:10,037 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/c28242ffaa9c4403a66f5856b794d48a, entries=150, sequenceid=448, filesize=11.9 K 2023-04-21 01:55:10,038 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~285.33 KB/292176, currentSize=80.51 KB/82440 for 65bad43ef0e690f33e72a230ffd7ee4f in 849ms, sequenceid=448, compaction requested=true 2023-04-21 01:55:10,038 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:10,038 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:10,038 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:10,038 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:10,038 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:10,038 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:10,039 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:10,039 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39714 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:10,039 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/B is initiating minor compaction (all files) 2023-04-21 01:55:10,040 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/B in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:10,040 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/d0584eb9365a42faa0b974d8b46fd174, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/051f571ec38249d385386b72770f33a0, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/a64fae16e7194454b0823a76701ba4f0] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=36.4 K 2023-04-21 01:55:10,039 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/A is initiating minor compaction (all files) 2023-04-21 01:55:10,040 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting d0584eb9365a42faa0b974d8b46fd174, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1682042105838 2023-04-21 01:55:10,040 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/A in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:10,040 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/894ec71e5bad4ed3ae0136e2eccff7a4, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/2ffb55d109d1429e88c57de62c3d5d85, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/bd7b9ce40ca4413aab9bc90f166c6757] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=38.8 K 2023-04-21 01:55:10,040 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 051f571ec38249d385386b72770f33a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1682042106981 2023-04-21 01:55:10,041 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 894ec71e5bad4ed3ae0136e2eccff7a4, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1682042105838 2023-04-21 01:55:10,041 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting a64fae16e7194454b0823a76701ba4f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1682042109167 2023-04-21 01:55:10,041 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 2ffb55d109d1429e88c57de62c3d5d85, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1682042106981 2023-04-21 01:55:10,042 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting bd7b9ce40ca4413aab9bc90f166c6757, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1682042109167 2023-04-21 01:55:10,047 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#B#compaction#306 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:10,048 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#A#compaction#307 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:10,060 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/eb3462eb006f4928b9bbf1f388582efc as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/eb3462eb006f4928b9bbf1f388582efc 2023-04-21 01:55:10,060 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/d685528bfc2f41ef91f058c6b50c93d6 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/d685528bfc2f41ef91f058c6b50c93d6 2023-04-21 01:55:10,064 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/A of 65bad43ef0e690f33e72a230ffd7ee4f into d685528bfc2f41ef91f058c6b50c93d6(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:10,064 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:10,064 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/A, priority=13, startTime=1682042110038; duration=0sec 2023-04-21 01:55:10,064 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:10,065 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:10,065 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/B of 65bad43ef0e690f33e72a230ffd7ee4f into eb3462eb006f4928b9bbf1f388582efc(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:10,065 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:10,065 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/B, priority=13, startTime=1682042110038; duration=0sec 2023-04-21 01:55:10,065 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:10,065 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:10,065 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/C is initiating minor compaction (all files) 2023-04-21 01:55:10,066 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/C in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:10,066 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/2615a761556840eba764eedd8ab8ffcc, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/ebb0c3c3c03246d2a731d09d09f0515e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/c28242ffaa9c4403a66f5856b794d48a] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=36.4 K 2023-04-21 01:55:10,066 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 2615a761556840eba764eedd8ab8ffcc, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1682042105838 2023-04-21 01:55:10,066 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting ebb0c3c3c03246d2a731d09d09f0515e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1682042106981 2023-04-21 01:55:10,066 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting c28242ffaa9c4403a66f5856b794d48a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1682042109167 2023-04-21 01:55:10,071 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:10,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:10,073 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#C#compaction#308 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:10,073 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:10,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:10,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:10,074 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:10,074 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:10,074 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:10,074 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:10,075 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:10,075 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:10,077 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:10,078 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:10,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:10,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:10,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:10,078 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:10,078 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:10,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:10,082 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/ede5d9b8cf584042b27dd31c1cdf3d1a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/ede5d9b8cf584042b27dd31c1cdf3d1a 2023-04-21 01:55:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:10,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:10,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:10,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:10,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:10,083 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:10,083 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:10,083 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:10,083 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:10,083 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:10,083 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:10,083 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:10,083 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-21 01:55:10,084 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:55:10,084 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:10,084 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:55:10,084 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:10,084 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:55:10,084 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:10,087 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/C of 65bad43ef0e690f33e72a230ffd7ee4f into ede5d9b8cf584042b27dd31c1cdf3d1a(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:10,087 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:10,087 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/C, priority=13, startTime=1682042110038; duration=0sec 2023-04-21 01:55:10,087 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:10,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:55:10,351 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:10,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:10,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 399 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042170380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:10,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:10,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 401 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042170482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:10,491 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/bf2a4fb5b7c6406bae5ad86ae3a2655d 2023-04-21 01:55:10,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:10,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 403 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042170686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:10,901 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/b77bafde3ab54d8e81a20cbd2b9ab118 2023-04-21 01:55:10,911 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/19cd4b67c1264a329fa703982d05188a 2023-04-21 01:55:10,915 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/bf2a4fb5b7c6406bae5ad86ae3a2655d as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/bf2a4fb5b7c6406bae5ad86ae3a2655d 2023-04-21 01:55:10,918 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/bf2a4fb5b7c6406bae5ad86ae3a2655d, entries=150, sequenceid=465, filesize=11.9 K 2023-04-21 01:55:10,919 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/b77bafde3ab54d8e81a20cbd2b9ab118 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/b77bafde3ab54d8e81a20cbd2b9ab118 2023-04-21 01:55:10,922 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/b77bafde3ab54d8e81a20cbd2b9ab118, entries=150, sequenceid=465, filesize=11.9 K 2023-04-21 01:55:10,923 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/19cd4b67c1264a329fa703982d05188a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/19cd4b67c1264a329fa703982d05188a 2023-04-21 01:55:10,926 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/19cd4b67c1264a329fa703982d05188a, entries=150, sequenceid=465, filesize=11.9 K 2023-04-21 01:55:10,927 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 65bad43ef0e690f33e72a230ffd7ee4f in 844ms, sequenceid=465, compaction requested=false 2023-04-21 01:55:10,927 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:10,927 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:10,927 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:10,927 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:10,927 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:10,931 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:10,931 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:10,933 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:10,933 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:10,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:10,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:10,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:10,934 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:10,934 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:10,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:10,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:10,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:10,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:10,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:10,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:10,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@82b46a4[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:10,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:10,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:10,940 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:10,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:10,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:10,941 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:10,941 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:10,941 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:10,941 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:10,942 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:10,942 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:10,942 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:10,942 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:10,943 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:10,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:10,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:10,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:10,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:10,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:10,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:10,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:10,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:10,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:10,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:10,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:10,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:10,945 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:10,945 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:10,945 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:10,945 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:10,946 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:10,949 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:10,949 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:10,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:10,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:10,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:10,949 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:10,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:10,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:10,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:10,950 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:10,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:10,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:10,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:10,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:10,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:10,951 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:10,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:10,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:10,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:10,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:10,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:10,953 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:10,953 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:10,953 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:10,953 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:10,953 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:10,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:10,953 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:10,953 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:10,953 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:10,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:10,953 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:10,954 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:10,954 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:10,954 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:10,954 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:10,954 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:10,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:10,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:10,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:10,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:55:10,991 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-21 01:55:10,991 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:55:10,991 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:10,991 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:55:10,991 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:10,991 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:55:10,991 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:11,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:11,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52802 deadline: 1682042171012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:11,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 415 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042171013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,054 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:11,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:11,056 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:11,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:11,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:11,057 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:11,057 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:11,057 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:11,057 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:11,058 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,058 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,059 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:11,059 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:11,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:11,059 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,059 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:11,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:11,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,060 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:11,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:11,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:11,060 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:11,061 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:11,061 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:11,061 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:11,061 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:11,061 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:11,061 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:11,061 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:11,061 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:11,061 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:11,061 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:11,061 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,061 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:11,061 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:11,063 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,063 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:11,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:11,063 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,063 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:11,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:11,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:11,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:11,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:11,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@666fee31[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:11,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:11,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,068 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:11,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:11,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:11,068 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,068 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:11,069 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:11,069 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,069 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:11,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:11,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:11,069 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:11,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:11,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:11,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:11,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:11,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:11,072 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:11,072 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:11,072 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:11,072 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:11,072 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:11,074 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,074 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:11,074 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:11,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:11,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:11,079 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:11,081 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:11,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,086 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,087 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,087 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,087 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,087 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,087 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,087 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,087 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:11,087 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:11,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:11,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:11,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:11,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:11,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:11,093 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:11,093 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:11,094 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:11,094 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:11,094 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:11,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:11,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52802 deadline: 1682042171115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:11,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 417 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042171116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,194 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:11,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:11,196 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:11,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:11,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:11,197 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:11,197 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:11,197 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:11,197 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:11,198 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,198 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,199 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:11,199 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:11,200 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,200 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:11,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:11,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:11,201 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:11,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:11,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:11,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:11,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:11,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:11,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:11,201 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:11,201 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:11,201 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:11,201 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:11,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:11,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:11,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:11,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:11,203 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,203 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:11,203 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:11,203 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:11,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:11,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:11,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d49501f[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:11,205 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:11,205 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,206 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,206 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:11,206 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,206 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:11,206 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:11,206 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:11,208 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,208 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:11,208 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:11,208 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:11,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:11,211 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:11,211 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:11,211 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:11,211 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:11,211 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:11,212 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:11,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:11,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:11,213 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:11,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,214 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,216 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,216 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:11,216 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:11,216 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,216 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,216 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,216 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,216 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,216 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,216 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,216 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:11,217 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:11,217 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:11,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:11,217 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:11,217 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:11,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:11,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:11,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:11,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:11,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:11,318 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:11,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:11,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:11,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52802 deadline: 1682042171318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,320 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:11,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:11,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 419 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042171320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:11,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:11,321 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:11,321 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:11,321 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:11,321 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:11,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,323 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:11,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:11,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:11,323 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,324 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,324 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:11,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,325 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:11,325 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:11,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:11,325 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:11,325 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:11,325 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:11,326 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:11,326 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:11,326 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:11,326 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:11,326 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:11,326 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:11,326 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:11,326 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:11,326 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,326 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:11,326 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:11,329 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,329 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:11,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:11,329 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,329 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:11,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:11,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:11,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:11,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:11,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57f2f630[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:11,330 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:11,330 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,332 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,332 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:11,332 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:11,332 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:11,332 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,332 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:11,333 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,333 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:11,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:11,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:11,333 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:11,334 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:11,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:11,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:11,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:11,337 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:11,337 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:11,337 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:11,337 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:11,337 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:11,339 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,339 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:11,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,339 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:11,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:11,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:11,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:11,339 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:11,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:11,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,340 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,344 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:11,344 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:11,344 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,344 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,344 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:11,344 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,344 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,344 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:11,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:11,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:11,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:11,344 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:11,345 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:11,344 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,345 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,345 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:11,345 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:11,399 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/2e5046b4830e4fe282b6725829eb3de8 2023-04-21 01:55:11,410 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/0c42cecf9bf647be835804e7d5b3f43a 2023-04-21 01:55:11,420 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/aba7b2f10d794995843c21aa9a99a932 2023-04-21 01:55:11,423 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/2e5046b4830e4fe282b6725829eb3de8 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/2e5046b4830e4fe282b6725829eb3de8 2023-04-21 01:55:11,427 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/2e5046b4830e4fe282b6725829eb3de8, entries=150, sequenceid=484, filesize=11.9 K 2023-04-21 01:55:11,427 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/0c42cecf9bf647be835804e7d5b3f43a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/0c42cecf9bf647be835804e7d5b3f43a 2023-04-21 01:55:11,431 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/0c42cecf9bf647be835804e7d5b3f43a, entries=150, sequenceid=484, filesize=11.9 K 2023-04-21 01:55:11,432 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/aba7b2f10d794995843c21aa9a99a932 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/aba7b2f10d794995843c21aa9a99a932 2023-04-21 01:55:11,435 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/aba7b2f10d794995843c21aa9a99a932, entries=150, sequenceid=484, filesize=11.9 K 2023-04-21 01:55:11,436 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 65bad43ef0e690f33e72a230ffd7ee4f in 445ms, sequenceid=484, compaction requested=true 2023-04-21 01:55:11,436 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:11,436 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:11,436 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:11,436 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:11,436 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:11,436 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:11,437 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:11,437 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/A is initiating minor compaction (all files) 2023-04-21 01:55:11,437 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/A in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:11,437 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/d685528bfc2f41ef91f058c6b50c93d6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/bf2a4fb5b7c6406bae5ad86ae3a2655d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/2e5046b4830e4fe282b6725829eb3de8] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=36.5 K 2023-04-21 01:55:11,438 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:11,438 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/B is initiating minor compaction (all files) 2023-04-21 01:55:11,438 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/B in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:11,438 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/eb3462eb006f4928b9bbf1f388582efc, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/b77bafde3ab54d8e81a20cbd2b9ab118, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/0c42cecf9bf647be835804e7d5b3f43a] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=36.5 K 2023-04-21 01:55:11,438 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting d685528bfc2f41ef91f058c6b50c93d6, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1682042109167 2023-04-21 01:55:11,438 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting eb3462eb006f4928b9bbf1f388582efc, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1682042109167 2023-04-21 01:55:11,438 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting bf2a4fb5b7c6406bae5ad86ae3a2655d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1682042109221 2023-04-21 01:55:11,438 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting b77bafde3ab54d8e81a20cbd2b9ab118, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1682042109221 2023-04-21 01:55:11,439 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 2e5046b4830e4fe282b6725829eb3de8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682042110361 2023-04-21 01:55:11,439 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 0c42cecf9bf647be835804e7d5b3f43a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682042110361 2023-04-21 01:55:11,445 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:11,445 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#A#compaction#315 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:11,445 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#B#compaction#316 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:11,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:11,450 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:11,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:11,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:11,452 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:11,452 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:11,452 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:11,452 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:11,453 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,453 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,454 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:11,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:11,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:11,454 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,455 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,455 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:11,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:11,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:11,455 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:11,455 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:11,455 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:11,456 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:11,456 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:11,456 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:11,456 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:11,456 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-21 01:55:11,457 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:55:11,457 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:11,457 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:55:11,457 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:11,457 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:55:11,457 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:11,457 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/8137c9321b3240ff8650e51e10bee298 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/8137c9321b3240ff8650e51e10bee298 2023-04-21 01:55:11,461 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/A of 65bad43ef0e690f33e72a230ffd7ee4f into 8137c9321b3240ff8650e51e10bee298(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:11,462 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:11,462 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/A, priority=13, startTime=1682042111436; duration=0sec 2023-04-21 01:55:11,462 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:11,462 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:11,463 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:11,463 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/C is initiating minor compaction (all files) 2023-04-21 01:55:11,463 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/C in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:11,463 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/ede5d9b8cf584042b27dd31c1cdf3d1a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/19cd4b67c1264a329fa703982d05188a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/aba7b2f10d794995843c21aa9a99a932] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=36.5 K 2023-04-21 01:55:11,463 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting ede5d9b8cf584042b27dd31c1cdf3d1a, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1682042109167 2023-04-21 01:55:11,464 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 19cd4b67c1264a329fa703982d05188a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1682042109221 2023-04-21 01:55:11,464 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting aba7b2f10d794995843c21aa9a99a932, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682042110361 2023-04-21 01:55:11,470 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=498 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/55934b5de6644893a2c632e8ab2f2b83 2023-04-21 01:55:11,471 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#C#compaction#318 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:11,485 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=498 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/66dc48fabbd44bada4d99f276b0c9835 2023-04-21 01:55:11,498 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=498 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/a47523874a6142cf80d31ef832eea3c9 2023-04-21 01:55:11,503 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/55934b5de6644893a2c632e8ab2f2b83 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/55934b5de6644893a2c632e8ab2f2b83 2023-04-21 01:55:11,506 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/55934b5de6644893a2c632e8ab2f2b83, entries=150, sequenceid=498, filesize=11.9 K 2023-04-21 01:55:11,507 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/66dc48fabbd44bada4d99f276b0c9835 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/66dc48fabbd44bada4d99f276b0c9835 2023-04-21 01:55:11,510 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/66dc48fabbd44bada4d99f276b0c9835, entries=150, sequenceid=498, filesize=11.9 K 2023-04-21 01:55:11,511 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/a47523874a6142cf80d31ef832eea3c9 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a47523874a6142cf80d31ef832eea3c9 2023-04-21 01:55:11,514 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a47523874a6142cf80d31ef832eea3c9, entries=150, sequenceid=498, filesize=11.9 K 2023-04-21 01:55:11,515 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 65bad43ef0e690f33e72a230ffd7ee4f in 59ms, sequenceid=498, compaction requested=false 2023-04-21 01:55:11,515 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:11,515 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:11,515 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:11,515 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:11,515 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,515 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:11,515 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:11,517 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:11,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:11,517 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:11,517 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,517 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:11,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:11,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:11,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:11,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53e940c[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:11,518 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:11,518 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,520 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:11,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:11,520 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:11,520 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,520 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:11,522 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,523 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:11,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:11,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:11,523 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:11,523 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:11,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:11,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:11,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:11,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:11,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:11,525 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:11,525 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:11,525 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:11,525 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:11,525 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:11,527 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,527 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:11,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:11,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,527 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:11,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:11,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:11,527 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:11,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:11,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,529 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:11,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,533 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,533 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:11,533 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:11,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:11,533 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:11,533 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:11,533 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,533 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:11,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:11,533 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:11,533 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:11,533 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:11,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:11,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:11,533 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,534 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,534 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,534 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:55:11,631 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-21 01:55:11,631 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:55:11,631 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:11,631 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:55:11,631 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:11,631 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:55:11,631 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:11,633 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:11,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:11,636 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:11,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:11,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:11,637 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:11,637 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:11,637 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:11,637 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:11,638 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,638 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,640 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:11,640 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:11,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:11,641 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,641 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:11,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:11,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:11,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:11,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:11,642 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:11,642 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:11,642 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:11,642 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:11,642 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:11,642 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:11,642 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:11,642 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:11,642 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:11,642 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:11,642 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:11,642 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,643 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:11,643 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:11,644 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,644 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:11,644 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,644 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:11,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:11,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:11,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:11,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cb9ecfc[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:11,646 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:11,646 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,647 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,647 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:11,647 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,647 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:11,647 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:11,647 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:11,649 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,649 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:11,649 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:11,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:11,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:11,649 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:11,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:11,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:11,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:11,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:11,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:11,651 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:11,651 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:11,651 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:11,651 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:11,651 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:11,654 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,654 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:11,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:11,654 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:11,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:11,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:11,654 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:11,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:11,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,656 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:11,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,658 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:11,658 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:11,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:11,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:11,658 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,658 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,659 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,659 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:11,659 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,659 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:11,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:11,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:11,659 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,659 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,659 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,659 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:11,659 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:11,659 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:11,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042111659, max timestamp=1682042111669Num uniques 859; 2023-04-21 01:55:11,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042111659, max timestamp=1682042111669Num uniques 859; 2023-04-21 01:55:11,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:11,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042111659, max timestamp=1682042111669Num uniques 859; 2023-04-21 01:55:11,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:11,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:11,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:11,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 439 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042171697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:11,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52802 deadline: 1682042171700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,760 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:11,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:11,762 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:11,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:11,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:11,763 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:11,763 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:11,763 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:11,763 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:11,764 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,764 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,765 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,765 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:11,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:11,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:11,765 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,765 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:11,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:11,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:11,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:11,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:11,767 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:11,767 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:11,767 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:11,767 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:11,767 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:11,767 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:11,767 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:11,767 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:11,767 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:11,767 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:11,767 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:11,767 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,767 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:11,767 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:11,769 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,769 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:11,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:11,769 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,769 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:11,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:11,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:11,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:11,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:11,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b279edc[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:11,770 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:11,770 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,773 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,773 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:11,773 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:11,773 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:11,773 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,773 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:11,774 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:11,774 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:11,774 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,774 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:11,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:11,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:11,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:11,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:11,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:11,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:11,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:11,777 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:11,777 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:11,777 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:11,777 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:11,777 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:11,778 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,778 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:11,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,778 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:11,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:11,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:11,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:11,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:11,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,780 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:11,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,781 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:11,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,783 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:11,783 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:11,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:11,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:11,783 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:11,783 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,784 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:11,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:11,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:11,783 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:11,784 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:11,784 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,784 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,784 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,784 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,784 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,784 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:11,785 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:11,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:11,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 441 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042171801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:11,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52802 deadline: 1682042171803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,858 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/a21e8fdc09b14c6197663ed158fb7d93 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/a21e8fdc09b14c6197663ed158fb7d93 2023-04-21 01:55:11,862 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/B of 65bad43ef0e690f33e72a230ffd7ee4f into a21e8fdc09b14c6197663ed158fb7d93(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:11,862 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:11,862 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/B, priority=13, startTime=1682042111436; duration=0sec 2023-04-21 01:55:11,862 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:11,879 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/ba0c146b3ce9467c9068ee6115e6af2b as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/ba0c146b3ce9467c9068ee6115e6af2b 2023-04-21 01:55:11,883 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/C of 65bad43ef0e690f33e72a230ffd7ee4f into ba0c146b3ce9467c9068ee6115e6af2b(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:11,883 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:11,883 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/C, priority=13, startTime=1682042111436; duration=0sec 2023-04-21 01:55:11,883 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:11,885 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:11,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:11,887 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:11,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:11,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:11,888 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:11,888 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:11,888 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:11,888 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:11,888 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,888 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,890 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,890 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:11,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:11,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:11,891 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,891 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:11,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:11,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,892 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:11,892 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:11,892 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:11,892 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:11,892 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:11,892 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:11,892 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:11,892 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:11,892 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:11,892 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:11,892 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:11,893 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:11,893 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:11,893 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:11,893 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,893 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:11,893 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:11,894 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,894 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,894 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:11,894 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:11,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:11,896 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:11,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b24b2fe[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:11,896 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,897 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,897 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,897 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:11,897 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:11,897 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:11,897 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:11,898 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,898 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:11,898 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:11,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:11,898 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:11,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:11,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:11,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:11,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:11,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:11,901 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:11,901 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:11,901 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:11,901 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:11,901 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:11,904 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,904 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:11,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:11,904 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:11,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,904 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:11,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:11,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:11,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:11,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,906 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,908 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,908 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:11,908 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:11,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:11,908 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:11,908 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:11,908 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:11,908 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:11,909 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:11,909 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,909 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:11,909 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:11,909 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:11,908 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:11,909 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:11,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:11,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:11,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:12,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:12,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 443 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042172003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,009 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:12,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:12,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:12,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52802 deadline: 1682042172007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,011 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:12,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:12,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:12,012 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:12,012 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:12,012 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:12,012 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:12,012 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,012 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:12,013 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,013 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:12,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:12,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:12,014 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,014 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:12,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:12,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:12,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:12,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:12,015 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:12,015 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:12,015 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:12,015 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:12,015 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:12,015 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:12,015 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:12,015 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:12,015 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:12,015 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:12,015 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:12,015 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:12,015 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:12,015 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,015 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:12,015 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:12,020 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:12,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:12,020 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:12,020 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,021 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:12,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:12,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:12,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:12,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a4bc962[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:12,022 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:12,022 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,023 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,023 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:12,023 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,023 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:12,023 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:12,023 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:12,024 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:12,024 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:12,024 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,025 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:12,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:12,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:12,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:12,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:12,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:12,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:12,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:12,027 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:12,027 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:12,027 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:12,027 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:12,027 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:12,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:12,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:12,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:12,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:12,029 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:12,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:12,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:12,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,031 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:12,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,034 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:12,034 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:12,034 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:12,034 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,034 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:12,034 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:12,034 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:12,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:12,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:12,034 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:12,034 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,034 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,034 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,034 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:12,034 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:12,034 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:12,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:12,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:12,039 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/ac724fc68053497491c5c6dcc13df397 2023-04-21 01:55:12,135 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:12,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:12,137 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:12,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:12,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:12,138 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:12,138 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:12,138 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:12,138 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:12,138 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,139 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:12,140 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:12,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:12,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:12,140 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:12,140 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,140 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:12,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:12,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:12,142 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:12,142 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:12,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:12,142 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:12,142 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:12,142 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:12,143 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:12,143 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:12,143 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:12,143 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:12,144 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:12,144 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:12,145 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:12,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,146 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:12,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:12,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:12,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:12,147 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:12,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29c614db[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:12,147 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,148 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,148 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:12,148 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:12,148 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:12,148 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,148 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:12,156 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,156 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:12,156 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:12,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:12,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:12,156 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:12,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:12,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:12,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:12,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:12,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:12,158 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:12,158 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:12,158 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:12,158 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:12,158 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:12,160 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,160 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:12,160 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:12,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,161 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,162 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:12,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,164 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:12,164 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:12,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:12,164 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:12,164 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,164 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:12,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:12,164 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:12,164 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:12,164 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:12,164 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:12,164 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:12,164 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,164 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:12,164 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,165 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,264 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:12,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:12,267 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:12,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:12,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:12,268 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:12,268 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:12,268 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:12,268 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:12,268 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,268 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:12,269 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,269 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:12,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:12,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:12,270 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,270 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:12,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:12,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:12,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:12,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:12,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:12,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:12,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:12,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:12,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:12,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:12,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:12,271 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:12,271 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:12,271 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:12,271 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:12,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:12,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:12,271 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,272 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:12,272 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:12,274 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,274 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:12,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:12,274 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,274 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:12,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:12,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:12,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ee0be41[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:12,276 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:12,276 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,277 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,277 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:12,277 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:12,277 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:12,277 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,277 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:12,278 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:12,278 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:12,278 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,278 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:12,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:12,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:12,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:12,281 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:12,281 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:12,281 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:12,281 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:12,281 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:12,282 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,282 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,282 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:12,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:12,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:12,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:12,283 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:12,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:12,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:12,284 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:12,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,287 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,288 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:12,288 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:12,288 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:12,288 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:12,288 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:12,288 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:12,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:12,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:12,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:12,288 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:12,288 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:12,288 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:12,288 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,288 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,288 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,288 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:12,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:12,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:12,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 445 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52752 deadline: 1682042172308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:12,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52802 deadline: 1682042172312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,388 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:12,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:12,390 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:12,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:12,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:12,391 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:12,391 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:12,392 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:12,392 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:12,392 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,392 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:12,393 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,393 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:12,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:12,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:12,394 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,394 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:12,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:12,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:12,395 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:12,395 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:12,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:12,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:12,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:12,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:12,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:12,395 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:12,395 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:12,395 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:12,395 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:12,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:12,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:12,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:12,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:12,397 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,397 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:12,397 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,397 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:12,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:12,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70fd5421[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:12,399 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:12,399 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,403 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,404 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:12,404 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,404 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:12,404 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:12,404 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:12,405 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:12,405 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,405 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:12,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:12,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:12,405 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:12,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:12,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:12,408 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:12,408 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:12,408 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:12,408 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:12,408 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:12,409 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,409 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:12,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:12,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,409 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:12,410 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:12,411 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:12,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:12,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,413 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,413 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:12,413 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:12,413 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,413 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,413 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:12,413 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,413 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:12,413 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:12,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:12,413 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:12,413 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:12,413 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:12,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:12,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:12,413 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:12,414 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:12,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:12,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:12,450 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/827ea5bbec5b4aba98a419f886eb8efc 2023-04-21 01:55:12,461 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/637499be3ca546d88e9b8c56184fdecc 2023-04-21 01:55:12,470 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/ac724fc68053497491c5c6dcc13df397 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/ac724fc68053497491c5c6dcc13df397 2023-04-21 01:55:12,473 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/ac724fc68053497491c5c6dcc13df397, entries=150, sequenceid=509, filesize=11.9 K 2023-04-21 01:55:12,474 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/827ea5bbec5b4aba98a419f886eb8efc as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/827ea5bbec5b4aba98a419f886eb8efc 2023-04-21 01:55:12,478 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/827ea5bbec5b4aba98a419f886eb8efc, entries=150, sequenceid=509, filesize=11.9 K 2023-04-21 01:55:12,479 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/637499be3ca546d88e9b8c56184fdecc as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/637499be3ca546d88e9b8c56184fdecc 2023-04-21 01:55:12,488 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/637499be3ca546d88e9b8c56184fdecc, entries=150, sequenceid=509, filesize=11.9 K 2023-04-21 01:55:12,489 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 65bad43ef0e690f33e72a230ffd7ee4f in 858ms, sequenceid=509, compaction requested=true 2023-04-21 01:55:12,489 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:12,489 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:12,489 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:12,489 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:12,489 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:12,489 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:12,492 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:12,492 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/A is initiating minor compaction (all files) 2023-04-21 01:55:12,492 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:12,492 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/A in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:12,492 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/B is initiating minor compaction (all files) 2023-04-21 01:55:12,492 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/8137c9321b3240ff8650e51e10bee298, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/55934b5de6644893a2c632e8ab2f2b83, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/ac724fc68053497491c5c6dcc13df397] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=36.6 K 2023-04-21 01:55:12,492 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/B in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:12,493 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/a21e8fdc09b14c6197663ed158fb7d93, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/66dc48fabbd44bada4d99f276b0c9835, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/827ea5bbec5b4aba98a419f886eb8efc] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=36.6 K 2023-04-21 01:55:12,493 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 8137c9321b3240ff8650e51e10bee298, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682042110361 2023-04-21 01:55:12,493 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting a21e8fdc09b14c6197663ed158fb7d93, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682042110361 2023-04-21 01:55:12,494 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 55934b5de6644893a2c632e8ab2f2b83, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=498, earliestPutTs=1682042111007 2023-04-21 01:55:12,494 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 66dc48fabbd44bada4d99f276b0c9835, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=498, earliestPutTs=1682042111007 2023-04-21 01:55:12,494 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting ac724fc68053497491c5c6dcc13df397, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1682042111625 2023-04-21 01:55:12,494 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 827ea5bbec5b4aba98a419f886eb8efc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1682042111625 2023-04-21 01:55:12,500 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#A#compaction#324 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:12,503 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#B#compaction#325 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:12,514 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:12,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:12,516 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:12,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:12,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:12,524 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:12,524 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:12,524 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:12,524 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:12,525 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/6b052bea8c24433395325c6e1f2648d1 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/6b052bea8c24433395325c6e1f2648d1 2023-04-21 01:55:12,525 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,525 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:12,527 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:12,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:12,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:12,527 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,528 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:12,528 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:12,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:12,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:12,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:12,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:12,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:12,530 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/A of 65bad43ef0e690f33e72a230ffd7ee4f into 6b052bea8c24433395325c6e1f2648d1(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:12,530 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:12,530 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/A, priority=13, startTime=1682042112489; duration=0sec 2023-04-21 01:55:12,530 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:12,530 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:12,530 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:12,530 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:12,530 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:12,530 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:12,530 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:12,531 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:12,531 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:12,531 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:12,531 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. started... 2023-04-21 01:55:12,531 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:12,531 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-21 01:55:12,531 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 65bad43ef0e690f33e72a230ffd7ee4f/C is initiating minor compaction (all files) 2023-04-21 01:55:12,531 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 65bad43ef0e690f33e72a230ffd7ee4f/C in TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:12,531 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/ba0c146b3ce9467c9068ee6115e6af2b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a47523874a6142cf80d31ef832eea3c9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/637499be3ca546d88e9b8c56184fdecc] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp, totalSize=36.6 K 2023-04-21 01:55:12,531 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:55:12,531 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:12,531 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:55:12,531 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:12,531 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:55:12,531 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:12,532 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting ba0c146b3ce9467c9068ee6115e6af2b, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682042110361 2023-04-21 01:55:12,532 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting a47523874a6142cf80d31ef832eea3c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=498, earliestPutTs=1682042111007 2023-04-21 01:55:12,533 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 637499be3ca546d88e9b8c56184fdecc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1682042111625 2023-04-21 01:55:12,541 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 65bad43ef0e690f33e72a230ffd7ee4f#C#compaction#327 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:12,545 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/b1938d478e1a4353ad06842d85286c12 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/b1938d478e1a4353ad06842d85286c12 2023-04-21 01:55:12,557 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/B of 65bad43ef0e690f33e72a230ffd7ee4f into b1938d478e1a4353ad06842d85286c12(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:12,557 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:12,557 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/B, priority=13, startTime=1682042112489; duration=0sec 2023-04-21 01:55:12,557 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:12,711 DEBUG [Thread-1877] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66ee0bd0 to 127.0.0.1:58383 2023-04-21 01:55:12,711 DEBUG [Thread-1877] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:55:12,711 DEBUG [Thread-1871] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08cd7cb0 to 127.0.0.1:58383 2023-04-21 01:55:12,711 DEBUG [Thread-1871] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:55:12,712 DEBUG [Thread-1879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b4658f8 to 127.0.0.1:58383 2023-04-21 01:55:12,712 DEBUG [Thread-1879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:55:12,712 DEBUG [Thread-1875] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x433f251d to 127.0.0.1:58383 2023-04-21 01:55:12,712 DEBUG [Thread-1873] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0ea1ca03 to 127.0.0.1:58383 2023-04-21 01:55:12,712 DEBUG [Thread-1875] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:55:12,712 DEBUG [Thread-1873] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:55:12,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:55:12,815 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. as already flushing 2023-04-21 01:55:12,815 DEBUG [Thread-1868] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6919b2a0 to 127.0.0.1:58383 2023-04-21 01:55:12,815 DEBUG [Thread-1868] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:55:12,818 DEBUG [Thread-1862] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c124820 to 127.0.0.1:58383 2023-04-21 01:55:12,819 DEBUG [Thread-1862] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:55:12,963 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=549 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/5cf6bc8a7b7d445bbc6a29925460c307 2023-04-21 01:55:12,966 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/f23d985725b644d1b4f6352011fd62ff as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/f23d985725b644d1b4f6352011fd62ff 2023-04-21 01:55:12,969 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 65bad43ef0e690f33e72a230ffd7ee4f/C of 65bad43ef0e690f33e72a230ffd7ee4f into f23d985725b644d1b4f6352011fd62ff(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:12,969 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:12,969 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f., storeName=65bad43ef0e690f33e72a230ffd7ee4f/C, priority=13, startTime=1682042112489; duration=0sec 2023-04-21 01:55:12,969 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:13,372 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=549 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/f8ccc88456d84a5c8868e663a281d2a9 2023-04-21 01:55:13,781 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=549 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/e6709d9d2f844ca3baf0833775b98f46 2023-04-21 01:55:13,785 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/5cf6bc8a7b7d445bbc6a29925460c307 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/5cf6bc8a7b7d445bbc6a29925460c307 2023-04-21 01:55:13,788 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/5cf6bc8a7b7d445bbc6a29925460c307, entries=150, sequenceid=549, filesize=11.9 K 2023-04-21 01:55:13,788 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/f8ccc88456d84a5c8868e663a281d2a9 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/f8ccc88456d84a5c8868e663a281d2a9 2023-04-21 01:55:13,791 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/f8ccc88456d84a5c8868e663a281d2a9, entries=150, sequenceid=549, filesize=11.9 K 2023-04-21 01:55:13,792 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/e6709d9d2f844ca3baf0833775b98f46 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/e6709d9d2f844ca3baf0833775b98f46 2023-04-21 01:55:13,795 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/e6709d9d2f844ca3baf0833775b98f46, entries=150, sequenceid=549, filesize=11.9 K 2023-04-21 01:55:13,796 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=13.42 KB/13740 for 65bad43ef0e690f33e72a230ffd7ee4f in 1265ms, sequenceid=549, compaction requested=false 2023-04-21 01:55:13,796 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:13,796 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:13,796 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:13,796 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:13,796 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:13,796 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:13,796 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:13,797 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:13,797 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:13,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:13,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:13,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:13,797 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:13,797 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:13,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:13,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:13,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:13,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:13,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:13,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:13,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54d98385[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:13,799 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:13,799 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:13,800 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:13,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:13,800 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:13,800 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:13,800 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:13,800 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:13,800 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:13,801 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:13,801 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:13,801 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:13,801 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:13,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:13,803 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:13,803 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:13,803 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:13,803 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:13,803 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:13,805 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:13,805 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:13,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:13,805 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:13,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:13,806 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:13,806 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:13,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:13,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:13,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:13,808 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:13,808 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:13,808 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:13,808 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:13,808 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:13,808 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:13,808 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:13,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:13,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:13,808 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:13,809 DEBUG [Thread-1870] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:13,808 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:13,809 DEBUG [Thread-1870] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:13,809 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:13,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:13,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:13,809 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:13,809 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:13,909 DEBUG [Thread-1870] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:13,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:15,993 DEBUG [Thread-1860] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23c97092 to 127.0.0.1:58383 2023-04-21 01:55:15,993 DEBUG [Thread-1860] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:55:16,034 DEBUG [Thread-1864] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3157d4df to 127.0.0.1:58383 2023-04-21 01:55:16,034 DEBUG [Thread-1864] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:55:16,059 DEBUG [Thread-1866] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cf5bdb7 to 127.0.0.1:58383 2023-04-21 01:55:16,059 DEBUG [Thread-1866] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:55:16,059 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-21 01:55:16,059 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 54 2023-04-21 01:55:16,059 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 38 2023-04-21 01:55:16,059 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 28 2023-04-21 01:55:16,059 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 28 2023-04-21 01:55:16,059 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 310 2023-04-21 01:55:16,060 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-21 01:55:16,060 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(398): read 5643 2023-04-21 01:55:16,060 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(398): read 5574 2023-04-21 01:55:16,060 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(398): read 5413 2023-04-21 01:55:16,060 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(398): read 5607 2023-04-21 01:55:16,060 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(398): read 5558 2023-04-21 01:55:16,060 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-21 01:55:16,060 INFO [Listener at localhost/39939] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-21 01:55:16,060 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12e82edd to 127.0.0.1:58383 2023-04-21 01:55:16,060 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:55:16,060 INFO [Listener at localhost/39939] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-21 01:55:16,061 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-21 01:55:16,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-21 01:55:16,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-21 01:55:16,064 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042116064"}]},"ts":"1682042116064"} 2023-04-21 01:55:16,065 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-21 01:55:16,066 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-21 01:55:16,067 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=65bad43ef0e690f33e72a230ffd7ee4f, UNASSIGN}] 2023-04-21 01:55:16,068 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=65bad43ef0e690f33e72a230ffd7ee4f, UNASSIGN 2023-04-21 01:55:16,069 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=65bad43ef0e690f33e72a230ffd7ee4f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:16,069 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042116069"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682042116069"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682042116069"}]},"ts":"1682042116069"} 2023-04-21 01:55:16,070 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399}] 2023-04-21 01:55:16,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-21 01:55:16,221 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:55:16,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 65bad43ef0e690f33e72a230ffd7ee4f, disabling compactions & flushes 2023-04-21 01:55:16,221 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:16,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:16,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. after waiting 0 ms 2023-04-21 01:55:16,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:16,221 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 65bad43ef0e690f33e72a230ffd7ee4f 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-21 01:55:16,222 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=A 2023-04-21 01:55:16,222 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:16,222 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=B 2023-04-21 01:55:16,222 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:16,222 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 65bad43ef0e690f33e72a230ffd7ee4f, store=C 2023-04-21 01:55:16,222 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:16,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-21 01:55:16,631 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/ae2d85a952a0491aadbdb2db216dff18 2023-04-21 01:55:16,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-21 01:55:17,040 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/621fb1db89d44f9da3d90cd56000aa98 2023-04-21 01:55:17,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-21 01:55:17,448 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/f204f13f4aa745d88f7f6843cdec39fd 2023-04-21 01:55:17,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/A/ae2d85a952a0491aadbdb2db216dff18 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/ae2d85a952a0491aadbdb2db216dff18 2023-04-21 01:55:17,456 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/ae2d85a952a0491aadbdb2db216dff18, entries=150, sequenceid=559, filesize=11.9 K 2023-04-21 01:55:17,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/B/621fb1db89d44f9da3d90cd56000aa98 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/621fb1db89d44f9da3d90cd56000aa98 2023-04-21 01:55:17,460 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/621fb1db89d44f9da3d90cd56000aa98, entries=150, sequenceid=559, filesize=11.9 K 2023-04-21 01:55:17,461 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/.tmp/C/f204f13f4aa745d88f7f6843cdec39fd as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/f204f13f4aa745d88f7f6843cdec39fd 2023-04-21 01:55:17,464 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/f204f13f4aa745d88f7f6843cdec39fd, entries=150, sequenceid=559, filesize=11.9 K 2023-04-21 01:55:17,465 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 65bad43ef0e690f33e72a230ffd7ee4f in 1244ms, sequenceid=559, compaction requested=true 2023-04-21 01:55:17,485 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/c0067f0b089e4a189c6241100fd87dee, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/b6e9b995e479405b82455e914f7c0ead, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/3dd5c55c600849889a96441959c441d2, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/014047aa59ec4fdba73ef1bcace58ea9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/962563355d2c4759991cfb3595c322d3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/df19bee729864e1080e2e31093941a78, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/040d73a3fd8941f0bfc43c789e927ddc, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/c0d51c47d69f41b3aa3bde6407d60a92, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/9caa7e1f78f248e381e0d9d7ab430836, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/725934dcc21649619e4b80953aeb940f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/823ca5dc7eb243c49e74d44c60711cf3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/311fd1aa775d4fe885096f374ef33c1b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/8aee80d3205b48dfa766bf162416878a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/04130290fbc743b496ea61d17f2f8f98, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/008f70626533407db9586fb727ed1e24, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/7553b1bc471c43fd97f2193bc8a4bfd5, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/cd489df9b6954ac7aaaf695ab36c76ce, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/864b6ef7ea6b4dc680db9c1c578317a3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/7781c36b19404e8cb991688f9004ced4, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/7a0392f7d93a435b84bb4d752ae61545, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/9c6034905a364c2093c670490aab92f7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/894ec71e5bad4ed3ae0136e2eccff7a4, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/2ffb55d109d1429e88c57de62c3d5d85, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/bd7b9ce40ca4413aab9bc90f166c6757, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/d685528bfc2f41ef91f058c6b50c93d6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/bf2a4fb5b7c6406bae5ad86ae3a2655d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/8137c9321b3240ff8650e51e10bee298, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/2e5046b4830e4fe282b6725829eb3de8, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/55934b5de6644893a2c632e8ab2f2b83, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/ac724fc68053497491c5c6dcc13df397] to archive 2023-04-21 01:55:17,486 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-21 01:55:17,488 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/c0067f0b089e4a189c6241100fd87dee to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/c0067f0b089e4a189c6241100fd87dee 2023-04-21 01:55:17,489 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/b6e9b995e479405b82455e914f7c0ead to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/b6e9b995e479405b82455e914f7c0ead 2023-04-21 01:55:17,490 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/3dd5c55c600849889a96441959c441d2 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/3dd5c55c600849889a96441959c441d2 2023-04-21 01:55:17,491 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/014047aa59ec4fdba73ef1bcace58ea9 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/014047aa59ec4fdba73ef1bcace58ea9 2023-04-21 01:55:17,492 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/962563355d2c4759991cfb3595c322d3 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/962563355d2c4759991cfb3595c322d3 2023-04-21 01:55:17,494 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/df19bee729864e1080e2e31093941a78 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/df19bee729864e1080e2e31093941a78 2023-04-21 01:55:17,494 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/040d73a3fd8941f0bfc43c789e927ddc to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/040d73a3fd8941f0bfc43c789e927ddc 2023-04-21 01:55:17,495 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/c0d51c47d69f41b3aa3bde6407d60a92 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/c0d51c47d69f41b3aa3bde6407d60a92 2023-04-21 01:55:17,496 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/9caa7e1f78f248e381e0d9d7ab430836 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/9caa7e1f78f248e381e0d9d7ab430836 2023-04-21 01:55:17,497 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/725934dcc21649619e4b80953aeb940f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/725934dcc21649619e4b80953aeb940f 2023-04-21 01:55:17,498 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/823ca5dc7eb243c49e74d44c60711cf3 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/823ca5dc7eb243c49e74d44c60711cf3 2023-04-21 01:55:17,499 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/311fd1aa775d4fe885096f374ef33c1b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/311fd1aa775d4fe885096f374ef33c1b 2023-04-21 01:55:17,500 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/8aee80d3205b48dfa766bf162416878a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/8aee80d3205b48dfa766bf162416878a 2023-04-21 01:55:17,501 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/04130290fbc743b496ea61d17f2f8f98 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/04130290fbc743b496ea61d17f2f8f98 2023-04-21 01:55:17,502 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/008f70626533407db9586fb727ed1e24 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/008f70626533407db9586fb727ed1e24 2023-04-21 01:55:17,503 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/7553b1bc471c43fd97f2193bc8a4bfd5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/7553b1bc471c43fd97f2193bc8a4bfd5 2023-04-21 01:55:17,504 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/cd489df9b6954ac7aaaf695ab36c76ce to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/cd489df9b6954ac7aaaf695ab36c76ce 2023-04-21 01:55:17,505 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/864b6ef7ea6b4dc680db9c1c578317a3 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/864b6ef7ea6b4dc680db9c1c578317a3 2023-04-21 01:55:17,506 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/7781c36b19404e8cb991688f9004ced4 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/7781c36b19404e8cb991688f9004ced4 2023-04-21 01:55:17,507 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/7a0392f7d93a435b84bb4d752ae61545 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/7a0392f7d93a435b84bb4d752ae61545 2023-04-21 01:55:17,507 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/9c6034905a364c2093c670490aab92f7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/9c6034905a364c2093c670490aab92f7 2023-04-21 01:55:17,509 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/894ec71e5bad4ed3ae0136e2eccff7a4 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/894ec71e5bad4ed3ae0136e2eccff7a4 2023-04-21 01:55:17,509 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/2ffb55d109d1429e88c57de62c3d5d85 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/2ffb55d109d1429e88c57de62c3d5d85 2023-04-21 01:55:17,510 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/bd7b9ce40ca4413aab9bc90f166c6757 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/bd7b9ce40ca4413aab9bc90f166c6757 2023-04-21 01:55:17,511 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/d685528bfc2f41ef91f058c6b50c93d6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/d685528bfc2f41ef91f058c6b50c93d6 2023-04-21 01:55:17,512 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/bf2a4fb5b7c6406bae5ad86ae3a2655d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/bf2a4fb5b7c6406bae5ad86ae3a2655d 2023-04-21 01:55:17,513 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/8137c9321b3240ff8650e51e10bee298 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/8137c9321b3240ff8650e51e10bee298 2023-04-21 01:55:17,514 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/2e5046b4830e4fe282b6725829eb3de8 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/2e5046b4830e4fe282b6725829eb3de8 2023-04-21 01:55:17,515 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/55934b5de6644893a2c632e8ab2f2b83 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/55934b5de6644893a2c632e8ab2f2b83 2023-04-21 01:55:17,516 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/ac724fc68053497491c5c6dcc13df397 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/ac724fc68053497491c5c6dcc13df397 2023-04-21 01:55:17,519 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/ca3d6f55b9374776b92127793f0275cd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/c2211c36751c4aa59fa22ee75bb280e5, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/9568a1a25dc9402ea6ed1503f59edcfa, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/7cd5c498969c4695b9210c5d3de804d1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/67d83b6c1d9341e38c0d44a9583431df, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/9c2ba92e4efe4e0496a6e88bf6fc853a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/3381dfa5347f47d788d0ea57a7a985b4, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/340b07ec39ec443e8f9709b53b05a89d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/6b569012bdb94b059ce154b348f30b55, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/abf5e1d39cf24e338a2d143f26c72b90, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/e40ce5703f3a4e87a540f86f841ad4c1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/f1e87e89c13646f18576173bd9d45724, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/51cda5dcdedf4ef7ae3b5f9d2b6f4145, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/dbb3054afe19435c88bf424d63b1de9e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/1a58373ef20741a7b40681733ec19f2c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/627e2933fb2f46598ba1254c164e1f2a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/dbfa142a52974e65997c72bd7b750a46, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/f29a5509c228496d8190c708adfa28d3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/0bf4217ea945471f9e3d48f7e9c9cc67, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/1aec52510c4f42fdb9c6818916233268, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/d0584eb9365a42faa0b974d8b46fd174, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/8d57cea147df416abf76ab84e5e7a8e5, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/051f571ec38249d385386b72770f33a0, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/eb3462eb006f4928b9bbf1f388582efc, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/a64fae16e7194454b0823a76701ba4f0, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/b77bafde3ab54d8e81a20cbd2b9ab118, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/a21e8fdc09b14c6197663ed158fb7d93, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/0c42cecf9bf647be835804e7d5b3f43a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/66dc48fabbd44bada4d99f276b0c9835, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/827ea5bbec5b4aba98a419f886eb8efc] to archive 2023-04-21 01:55:17,519 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-21 01:55:17,521 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/ca3d6f55b9374776b92127793f0275cd to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/ca3d6f55b9374776b92127793f0275cd 2023-04-21 01:55:17,522 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/c2211c36751c4aa59fa22ee75bb280e5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/c2211c36751c4aa59fa22ee75bb280e5 2023-04-21 01:55:17,522 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/9568a1a25dc9402ea6ed1503f59edcfa to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/9568a1a25dc9402ea6ed1503f59edcfa 2023-04-21 01:55:17,523 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/7cd5c498969c4695b9210c5d3de804d1 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/7cd5c498969c4695b9210c5d3de804d1 2023-04-21 01:55:17,524 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/67d83b6c1d9341e38c0d44a9583431df to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/67d83b6c1d9341e38c0d44a9583431df 2023-04-21 01:55:17,525 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/9c2ba92e4efe4e0496a6e88bf6fc853a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/9c2ba92e4efe4e0496a6e88bf6fc853a 2023-04-21 01:55:17,525 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/3381dfa5347f47d788d0ea57a7a985b4 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/3381dfa5347f47d788d0ea57a7a985b4 2023-04-21 01:55:17,526 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/340b07ec39ec443e8f9709b53b05a89d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/340b07ec39ec443e8f9709b53b05a89d 2023-04-21 01:55:17,527 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/6b569012bdb94b059ce154b348f30b55 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/6b569012bdb94b059ce154b348f30b55 2023-04-21 01:55:17,528 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/abf5e1d39cf24e338a2d143f26c72b90 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/abf5e1d39cf24e338a2d143f26c72b90 2023-04-21 01:55:17,529 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/e40ce5703f3a4e87a540f86f841ad4c1 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/e40ce5703f3a4e87a540f86f841ad4c1 2023-04-21 01:55:17,529 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/f1e87e89c13646f18576173bd9d45724 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/f1e87e89c13646f18576173bd9d45724 2023-04-21 01:55:17,530 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/51cda5dcdedf4ef7ae3b5f9d2b6f4145 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/51cda5dcdedf4ef7ae3b5f9d2b6f4145 2023-04-21 01:55:17,531 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/dbb3054afe19435c88bf424d63b1de9e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/dbb3054afe19435c88bf424d63b1de9e 2023-04-21 01:55:17,532 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/1a58373ef20741a7b40681733ec19f2c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/1a58373ef20741a7b40681733ec19f2c 2023-04-21 01:55:17,533 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/627e2933fb2f46598ba1254c164e1f2a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/627e2933fb2f46598ba1254c164e1f2a 2023-04-21 01:55:17,534 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/dbfa142a52974e65997c72bd7b750a46 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/dbfa142a52974e65997c72bd7b750a46 2023-04-21 01:55:17,534 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/f29a5509c228496d8190c708adfa28d3 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/f29a5509c228496d8190c708adfa28d3 2023-04-21 01:55:17,535 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/0bf4217ea945471f9e3d48f7e9c9cc67 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/0bf4217ea945471f9e3d48f7e9c9cc67 2023-04-21 01:55:17,536 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/1aec52510c4f42fdb9c6818916233268 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/1aec52510c4f42fdb9c6818916233268 2023-04-21 01:55:17,537 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/d0584eb9365a42faa0b974d8b46fd174 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/d0584eb9365a42faa0b974d8b46fd174 2023-04-21 01:55:17,538 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/8d57cea147df416abf76ab84e5e7a8e5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/8d57cea147df416abf76ab84e5e7a8e5 2023-04-21 01:55:17,538 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/051f571ec38249d385386b72770f33a0 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/051f571ec38249d385386b72770f33a0 2023-04-21 01:55:17,539 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/eb3462eb006f4928b9bbf1f388582efc to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/eb3462eb006f4928b9bbf1f388582efc 2023-04-21 01:55:17,540 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/a64fae16e7194454b0823a76701ba4f0 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/a64fae16e7194454b0823a76701ba4f0 2023-04-21 01:55:17,541 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/b77bafde3ab54d8e81a20cbd2b9ab118 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/b77bafde3ab54d8e81a20cbd2b9ab118 2023-04-21 01:55:17,542 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/a21e8fdc09b14c6197663ed158fb7d93 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/a21e8fdc09b14c6197663ed158fb7d93 2023-04-21 01:55:17,542 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/0c42cecf9bf647be835804e7d5b3f43a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/0c42cecf9bf647be835804e7d5b3f43a 2023-04-21 01:55:17,543 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/66dc48fabbd44bada4d99f276b0c9835 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/66dc48fabbd44bada4d99f276b0c9835 2023-04-21 01:55:17,544 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/827ea5bbec5b4aba98a419f886eb8efc to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/827ea5bbec5b4aba98a419f886eb8efc 2023-04-21 01:55:17,547 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/04933a20731f4bd29c1c513b40bf8e07, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/310a54e48c91461d8bac491a02a7a56a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/ef608552bce644c78fd173df6a833564, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/8ac454416e604d27bef5d30cf8296e03, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/d7b11eb3be664471b199add8b492fff2, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/e7f3325fa0904836bb38b0697fbce50b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/6ff13d8675cc4240a7a9bc8c9089e81e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/77258a14435f4ec98128954424ba5af3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/8e77c9befa53459580decfd699a6b091, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/f763d1b4ce9b465b86d794854eb8d611, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/f3060cac2bab492e9290bb96fa257907, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/8a5d066dfaf74159a57629cbf9bd7656, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/6147ebe5828e418499e07714cdfd153c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/2f424c26af1c40efb9a34ca8f0c311cc, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a3094ddd28854e1897129b10ac8628b0, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/56d718cf290a47d4940763286c27fff5, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/765ab96305ca4e5b80d030638c56b0da, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a2e04bbdd9ae47f8a156ed67705eaa55, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/8950d96fba414308893cee9524fd7fd6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a78803a333ea47b58535beb30011ce2d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/2615a761556840eba764eedd8ab8ffcc, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/4393802e223345588b1df85bc8215499, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/ebb0c3c3c03246d2a731d09d09f0515e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/ede5d9b8cf584042b27dd31c1cdf3d1a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/c28242ffaa9c4403a66f5856b794d48a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/19cd4b67c1264a329fa703982d05188a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/ba0c146b3ce9467c9068ee6115e6af2b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/aba7b2f10d794995843c21aa9a99a932, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a47523874a6142cf80d31ef832eea3c9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/637499be3ca546d88e9b8c56184fdecc] to archive 2023-04-21 01:55:17,548 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-21 01:55:17,549 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/04933a20731f4bd29c1c513b40bf8e07 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/04933a20731f4bd29c1c513b40bf8e07 2023-04-21 01:55:17,549 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/310a54e48c91461d8bac491a02a7a56a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/310a54e48c91461d8bac491a02a7a56a 2023-04-21 01:55:17,550 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/ef608552bce644c78fd173df6a833564 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/ef608552bce644c78fd173df6a833564 2023-04-21 01:55:17,551 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/8ac454416e604d27bef5d30cf8296e03 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/8ac454416e604d27bef5d30cf8296e03 2023-04-21 01:55:17,552 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/d7b11eb3be664471b199add8b492fff2 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/d7b11eb3be664471b199add8b492fff2 2023-04-21 01:55:17,553 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/e7f3325fa0904836bb38b0697fbce50b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/e7f3325fa0904836bb38b0697fbce50b 2023-04-21 01:55:17,554 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/6ff13d8675cc4240a7a9bc8c9089e81e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/6ff13d8675cc4240a7a9bc8c9089e81e 2023-04-21 01:55:17,555 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/77258a14435f4ec98128954424ba5af3 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/77258a14435f4ec98128954424ba5af3 2023-04-21 01:55:17,555 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/8e77c9befa53459580decfd699a6b091 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/8e77c9befa53459580decfd699a6b091 2023-04-21 01:55:17,556 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/f763d1b4ce9b465b86d794854eb8d611 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/f763d1b4ce9b465b86d794854eb8d611 2023-04-21 01:55:17,557 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/f3060cac2bab492e9290bb96fa257907 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/f3060cac2bab492e9290bb96fa257907 2023-04-21 01:55:17,558 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/8a5d066dfaf74159a57629cbf9bd7656 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/8a5d066dfaf74159a57629cbf9bd7656 2023-04-21 01:55:17,559 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/6147ebe5828e418499e07714cdfd153c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/6147ebe5828e418499e07714cdfd153c 2023-04-21 01:55:17,560 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/2f424c26af1c40efb9a34ca8f0c311cc to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/2f424c26af1c40efb9a34ca8f0c311cc 2023-04-21 01:55:17,561 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a3094ddd28854e1897129b10ac8628b0 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a3094ddd28854e1897129b10ac8628b0 2023-04-21 01:55:17,561 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/56d718cf290a47d4940763286c27fff5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/56d718cf290a47d4940763286c27fff5 2023-04-21 01:55:17,562 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/765ab96305ca4e5b80d030638c56b0da to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/765ab96305ca4e5b80d030638c56b0da 2023-04-21 01:55:17,563 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a2e04bbdd9ae47f8a156ed67705eaa55 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a2e04bbdd9ae47f8a156ed67705eaa55 2023-04-21 01:55:17,564 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/8950d96fba414308893cee9524fd7fd6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/8950d96fba414308893cee9524fd7fd6 2023-04-21 01:55:17,564 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a78803a333ea47b58535beb30011ce2d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a78803a333ea47b58535beb30011ce2d 2023-04-21 01:55:17,565 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/2615a761556840eba764eedd8ab8ffcc to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/2615a761556840eba764eedd8ab8ffcc 2023-04-21 01:55:17,566 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/4393802e223345588b1df85bc8215499 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/4393802e223345588b1df85bc8215499 2023-04-21 01:55:17,567 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/ebb0c3c3c03246d2a731d09d09f0515e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/ebb0c3c3c03246d2a731d09d09f0515e 2023-04-21 01:55:17,568 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/ede5d9b8cf584042b27dd31c1cdf3d1a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/ede5d9b8cf584042b27dd31c1cdf3d1a 2023-04-21 01:55:17,568 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/c28242ffaa9c4403a66f5856b794d48a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/c28242ffaa9c4403a66f5856b794d48a 2023-04-21 01:55:17,569 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/19cd4b67c1264a329fa703982d05188a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/19cd4b67c1264a329fa703982d05188a 2023-04-21 01:55:17,570 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/ba0c146b3ce9467c9068ee6115e6af2b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/ba0c146b3ce9467c9068ee6115e6af2b 2023-04-21 01:55:17,571 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/aba7b2f10d794995843c21aa9a99a932 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/aba7b2f10d794995843c21aa9a99a932 2023-04-21 01:55:17,572 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a47523874a6142cf80d31ef832eea3c9 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/a47523874a6142cf80d31ef832eea3c9 2023-04-21 01:55:17,573 DEBUG [StoreCloser-TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/637499be3ca546d88e9b8c56184fdecc to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/637499be3ca546d88e9b8c56184fdecc 2023-04-21 01:55:17,576 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/recovered.edits/562.seqid, newMaxSeqId=562, maxSeqId=1 2023-04-21 01:55:17,578 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f. 2023-04-21 01:55:17,578 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 65bad43ef0e690f33e72a230ffd7ee4f: 2023-04-21 01:55:17,579 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:55:17,580 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=65bad43ef0e690f33e72a230ffd7ee4f, regionState=CLOSED 2023-04-21 01:55:17,580 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042117580"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682042117580"}]},"ts":"1682042117580"} 2023-04-21 01:55:17,582 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-21 01:55:17,582 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 65bad43ef0e690f33e72a230ffd7ee4f, server=jenkins-hbase4.apache.org,40131,1682042019399 in 1.5110 sec 2023-04-21 01:55:17,583 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-21 01:55:17,583 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=65bad43ef0e690f33e72a230ffd7ee4f, UNASSIGN in 1.5150 sec 2023-04-21 01:55:17,584 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042117584"}]},"ts":"1682042117584"} 2023-04-21 01:55:17,584 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-21 01:55:17,586 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-21 01:55:17,587 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5250 sec 2023-04-21 01:55:18,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-21 01:55:18,168 INFO [Listener at localhost/39939] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-21 01:55:18,169 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-21 01:55:18,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:55:18,171 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:55:18,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-21 01:55:18,172 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:55:18,176 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:55:18,178 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A, FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B, FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C, FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/recovered.edits] 2023-04-21 01:55:18,180 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/5cf6bc8a7b7d445bbc6a29925460c307 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/5cf6bc8a7b7d445bbc6a29925460c307 2023-04-21 01:55:18,181 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/6b052bea8c24433395325c6e1f2648d1 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/6b052bea8c24433395325c6e1f2648d1 2023-04-21 01:55:18,182 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/ae2d85a952a0491aadbdb2db216dff18 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/A/ae2d85a952a0491aadbdb2db216dff18 2023-04-21 01:55:18,184 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/621fb1db89d44f9da3d90cd56000aa98 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/621fb1db89d44f9da3d90cd56000aa98 2023-04-21 01:55:18,185 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/b1938d478e1a4353ad06842d85286c12 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/b1938d478e1a4353ad06842d85286c12 2023-04-21 01:55:18,186 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/f8ccc88456d84a5c8868e663a281d2a9 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/B/f8ccc88456d84a5c8868e663a281d2a9 2023-04-21 01:55:18,188 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/e6709d9d2f844ca3baf0833775b98f46 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/e6709d9d2f844ca3baf0833775b98f46 2023-04-21 01:55:18,189 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/f204f13f4aa745d88f7f6843cdec39fd to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/f204f13f4aa745d88f7f6843cdec39fd 2023-04-21 01:55:18,190 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/f23d985725b644d1b4f6352011fd62ff to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/C/f23d985725b644d1b4f6352011fd62ff 2023-04-21 01:55:18,193 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/recovered.edits/562.seqid to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f/recovered.edits/562.seqid 2023-04-21 01:55:18,193 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/65bad43ef0e690f33e72a230ffd7ee4f 2023-04-21 01:55:18,193 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-21 01:55:18,195 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:55:18,197 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-21 01:55:18,198 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-21 01:55:18,199 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:55:18,199 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-21 01:55:18,199 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682042118199"}]},"ts":"9223372036854775807"} 2023-04-21 01:55:18,201 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-21 01:55:18,201 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 65bad43ef0e690f33e72a230ffd7ee4f, NAME => 'TestAcidGuarantees,,1682042090543.65bad43ef0e690f33e72a230ffd7ee4f.', STARTKEY => '', ENDKEY => ''}] 2023-04-21 01:55:18,201 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-21 01:55:18,201 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682042118201"}]},"ts":"9223372036854775807"} 2023-04-21 01:55:18,202 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-21 01:55:18,205 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:55:18,206 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 36 msec 2023-04-21 01:55:18,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-21 01:55:18,273 INFO [Listener at localhost/39939] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-21 01:55:18,284 INFO [Listener at localhost/39939] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=226 (was 224) - Thread LEAK? -, OpenFileDescriptor=569 (was 561) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=401 (was 484), ProcessCount=169 (was 169), AvailableMemoryMB=8651 (was 8676) 2023-04-21 01:55:18,295 INFO [Listener at localhost/39939] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=226, OpenFileDescriptor=569, MaxFileDescriptor=60000, SystemLoadAverage=401, ProcessCount=169, AvailableMemoryMB=8651 2023-04-21 01:55:18,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-21 01:55:18,298 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-21 01:55:18,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-21 01:55:18,300 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-21 01:55:18,300 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-21 01:55:18,301 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-21 01:55:18,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-21 01:55:18,302 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:18,302 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792 empty. 2023-04-21 01:55:18,303 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:18,303 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-21 01:55:18,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-21 01:55:18,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-21 01:55:18,709 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-21 01:55:18,711 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 810791807ce3c9f7e9f14187121bf792, NAME => 'TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp 2023-04-21 01:55:18,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-21 01:55:19,116 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-21 01:55:19,117 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 810791807ce3c9f7e9f14187121bf792, disabling compactions & flushes 2023-04-21 01:55:19,117 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:19,117 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:19,117 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. after waiting 0 ms 2023-04-21 01:55:19,117 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:19,117 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:19,117 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:19,119 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-21 01:55:19,120 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042119120"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682042119120"}]},"ts":"1682042119120"} 2023-04-21 01:55:19,121 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-21 01:55:19,121 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-21 01:55:19,121 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042119121"}]},"ts":"1682042119121"} 2023-04-21 01:55:19,122 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-21 01:55:19,124 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=810791807ce3c9f7e9f14187121bf792, ASSIGN}] 2023-04-21 01:55:19,126 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=810791807ce3c9f7e9f14187121bf792, ASSIGN 2023-04-21 01:55:19,126 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=810791807ce3c9f7e9f14187121bf792, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40131,1682042019399; forceNewPlan=false, retain=false 2023-04-21 01:55:19,277 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=810791807ce3c9f7e9f14187121bf792, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:19,278 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042119277"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682042119277"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682042119277"}]},"ts":"1682042119277"} 2023-04-21 01:55:19,279 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399}] 2023-04-21 01:55:19,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-21 01:55:19,433 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:19,433 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 810791807ce3c9f7e9f14187121bf792, NAME => 'TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.', STARTKEY => '', ENDKEY => ''} 2023-04-21 01:55:19,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:19,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-21 01:55:19,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:19,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:19,435 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:19,436 DEBUG [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A 2023-04-21 01:55:19,436 DEBUG [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A 2023-04-21 01:55:19,436 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:55:19,436 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 810791807ce3c9f7e9f14187121bf792 columnFamilyName A 2023-04-21 01:55:19,437 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] regionserver.HStore(310): Store=810791807ce3c9f7e9f14187121bf792/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:55:19,437 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:19,438 DEBUG [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B 2023-04-21 01:55:19,438 DEBUG [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B 2023-04-21 01:55:19,438 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:55:19,438 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 810791807ce3c9f7e9f14187121bf792 columnFamilyName B 2023-04-21 01:55:19,439 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] regionserver.HStore(310): Store=810791807ce3c9f7e9f14187121bf792/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:55:19,439 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:19,439 DEBUG [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C 2023-04-21 01:55:19,439 DEBUG [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C 2023-04-21 01:55:19,440 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:55:19,440 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 810791807ce3c9f7e9f14187121bf792 columnFamilyName C 2023-04-21 01:55:19,440 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] regionserver.HStore(310): Store=810791807ce3c9f7e9f14187121bf792/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:55:19,440 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:19,441 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:19,441 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:19,443 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-21 01:55:19,444 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:19,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-21 01:55:19,445 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 810791807ce3c9f7e9f14187121bf792; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64603579, jitterRate=-0.037331655621528625}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-21 01:55:19,446 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:19,446 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., pid=37, masterSystemTime=1682042119430 2023-04-21 01:55:19,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:19,447 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:19,448 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=810791807ce3c9f7e9f14187121bf792, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:19,448 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042119448"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682042119448"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682042119448"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682042119448"}]},"ts":"1682042119448"} 2023-04-21 01:55:19,450 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-21 01:55:19,450 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 in 170 msec 2023-04-21 01:55:19,451 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-21 01:55:19,451 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=810791807ce3c9f7e9f14187121bf792, ASSIGN in 326 msec 2023-04-21 01:55:19,451 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-21 01:55:19,452 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042119452"}]},"ts":"1682042119452"} 2023-04-21 01:55:19,452 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-21 01:55:19,454 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-21 01:55:19,455 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1560 sec 2023-04-21 01:55:20,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-21 01:55:20,406 INFO [Listener at localhost/39939] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-21 01:55:20,406 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x0fd3d54e to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:55:20,411 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6434ce29, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:55:20,412 DEBUG [Listener at localhost/39939] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:55:20,415 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38642, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:55:20,416 DEBUG [Listener at localhost/39939] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-21 01:55:20,417 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33886, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-21 01:55:20,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-21 01:55:20,418 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-21 01:55:20,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-21 01:55:20,829 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-21 01:55:20,830 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-21 01:55:20,830 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-21 01:55:20,832 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-21 01:55:20,834 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=810791807ce3c9f7e9f14187121bf792, REOPEN/MOVE}] 2023-04-21 01:55:20,835 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=810791807ce3c9f7e9f14187121bf792, REOPEN/MOVE 2023-04-21 01:55:20,835 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=810791807ce3c9f7e9f14187121bf792, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:20,836 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042120835"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682042120835"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682042120835"}]},"ts":"1682042120835"} 2023-04-21 01:55:20,836 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399}] 2023-04-21 01:55:20,988 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:20,989 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 810791807ce3c9f7e9f14187121bf792, disabling compactions & flushes 2023-04-21 01:55:20,989 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:20,989 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:20,989 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. after waiting 0 ms 2023-04-21 01:55:20,989 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:20,992 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-21 01:55:20,992 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:20,993 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:20,993 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 810791807ce3c9f7e9f14187121bf792 to self. 2023-04-21 01:55:20,994 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:20,994 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=810791807ce3c9f7e9f14187121bf792, regionState=CLOSED 2023-04-21 01:55:20,994 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042120994"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682042120994"}]},"ts":"1682042120994"} 2023-04-21 01:55:20,996 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-21 01:55:20,996 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 in 159 msec 2023-04-21 01:55:20,997 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=810791807ce3c9f7e9f14187121bf792, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,40131,1682042019399; forceNewPlan=false, retain=true 2023-04-21 01:55:21,147 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=810791807ce3c9f7e9f14187121bf792, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,148 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042121147"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682042121147"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682042121147"}]},"ts":"1682042121147"} 2023-04-21 01:55:21,149 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399}] 2023-04-21 01:55:21,304 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:21,304 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 810791807ce3c9f7e9f14187121bf792, NAME => 'TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.', STARTKEY => '', ENDKEY => ''} 2023-04-21 01:55:21,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:21,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-21 01:55:21,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:21,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:21,306 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:21,307 DEBUG [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A 2023-04-21 01:55:21,307 DEBUG [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A 2023-04-21 01:55:21,307 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:55:21,307 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 810791807ce3c9f7e9f14187121bf792 columnFamilyName A 2023-04-21 01:55:21,309 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] regionserver.HStore(310): Store=810791807ce3c9f7e9f14187121bf792/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:55:21,309 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:21,310 DEBUG [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B 2023-04-21 01:55:21,310 DEBUG [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B 2023-04-21 01:55:21,310 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:55:21,310 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 810791807ce3c9f7e9f14187121bf792 columnFamilyName B 2023-04-21 01:55:21,310 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] regionserver.HStore(310): Store=810791807ce3c9f7e9f14187121bf792/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:55:21,310 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:21,311 DEBUG [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C 2023-04-21 01:55:21,311 DEBUG [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C 2023-04-21 01:55:21,311 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:55:21,311 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 810791807ce3c9f7e9f14187121bf792 columnFamilyName C 2023-04-21 01:55:21,312 INFO [StoreOpener-810791807ce3c9f7e9f14187121bf792-1] regionserver.HStore(310): Store=810791807ce3c9f7e9f14187121bf792/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:55:21,312 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:21,313 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:21,313 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:21,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-21 01:55:21,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:21,317 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 810791807ce3c9f7e9f14187121bf792; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66293946, jitterRate=-0.012143224477767944}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-21 01:55:21,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:21,317 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., pid=42, masterSystemTime=1682042121300 2023-04-21 01:55:21,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:21,319 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:21,319 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=810791807ce3c9f7e9f14187121bf792, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,319 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042121319"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682042121319"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682042121319"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682042121319"}]},"ts":"1682042121319"} 2023-04-21 01:55:21,321 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-21 01:55:21,321 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 in 171 msec 2023-04-21 01:55:21,322 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-21 01:55:21,322 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=810791807ce3c9f7e9f14187121bf792, REOPEN/MOVE in 487 msec 2023-04-21 01:55:21,324 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-21 01:55:21,324 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 491 msec 2023-04-21 01:55:21,325 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 906 msec 2023-04-21 01:55:21,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-21 01:55:21,327 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x1a5ab074 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:55:21,331 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bb27fd4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:55:21,331 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x0bcfc1e0 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:55:21,335 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@445919fd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:55:21,336 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x38341134 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:55:21,341 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d010713, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:55:21,341 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x71b5058b to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:55:21,345 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69100738, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:55:21,346 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x3620ee35 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:55:21,352 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f603c27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:55:21,352 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x7e01d5e5 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:55:21,360 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10ee5f06, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:55:21,360 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x51ccd8b7 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:55:21,368 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21128196, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:55:21,368 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x68a0e21a to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:55:21,375 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9e86d9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:55:21,376 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x1ec12b02 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:55:21,380 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@711ebaf2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:55:21,380 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x1d7ba242 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:55:21,385 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a7503d4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:55:21,399 DEBUG [hconnection-0x6c544711-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:55:21,400 DEBUG [hconnection-0x3535a4f9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:55:21,400 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38654, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:55:21,408 DEBUG [hconnection-0x1599d37f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:55:21,408 DEBUG [hconnection-0x2bc07f6b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:55:21,408 DEBUG [hconnection-0xb7a7503-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:55:21,408 DEBUG [hconnection-0x7ec5f83a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:55:21,409 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38690, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:55:21,413 DEBUG [hconnection-0x2c29380a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:55:21,420 DEBUG [hconnection-0x1d809995-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:55:21,422 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38730, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:55:21,429 DEBUG [hconnection-0x111d228-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:55:21,435 DEBUG [hconnection-0x26bc8a5f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:55:21,436 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38664, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:55:21,436 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38674, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:55:21,436 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38700, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:55:21,437 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38742, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:55:21,437 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38758, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:55:21,436 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38716, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:55:21,439 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38752, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:55:21,447 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:21,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:21,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:21,460 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 810791807ce3c9f7e9f14187121bf792 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-21 01:55:21,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:21,460 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:21,460 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:21,460 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:21,460 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:21,461 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,461 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:21,462 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:21,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:21,462 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:21,462 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:21,462 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:21,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:21,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:21,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:21,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:21,464 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:21,464 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:21,464 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:21,468 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:21,468 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:21,468 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:21,468 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:21,468 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:21,468 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:21,468 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:21,468 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:21,468 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,468 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:21,468 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:21,473 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,473 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:21,473 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,473 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:21,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:21,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4da02509[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:21,474 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:21,474 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,475 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,475 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:21,475 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:21,476 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:21,476 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,476 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:21,477 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,477 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:21,477 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:21,477 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:21,477 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=A 2023-04-21 01:55:21,477 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:21,477 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=B 2023-04-21 01:55:21,477 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:21,477 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=C 2023-04-21 01:55:21,477 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:21,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:21,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:21,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:21,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:21,482 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:21,482 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:21,482 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:21,482 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:21,482 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:21,488 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,488 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,488 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:21,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:21,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:21,489 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,490 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:21,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:21,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682042181489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:21,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:21,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38730 deadline: 1682042181491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38700 deadline: 1682042181490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:21,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682042181491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042121487, max timestamp=1682042121490Num uniques 859; 2023-04-21 01:55:21,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042121487, max timestamp=1682042121490Num uniques 859; 2023-04-21 01:55:21,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:21,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042121487, max timestamp=1682042121490Num uniques 859; 2023-04-21 01:55:21,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:21,494 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,494 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:21,494 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:21,494 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:21,494 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:21,494 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:21,494 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:21,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:21,494 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:21,494 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,494 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,494 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:21,495 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:21,495 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:21,495 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:21,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38716 deadline: 1682042181508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:21,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682042181592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:21,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682042181592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:21,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38730 deadline: 1682042181592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:21,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38700 deadline: 1682042181593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,595 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:21,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:21,597 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:21,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:21,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:21,607 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:21,607 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:21,607 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:21,607 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:21,608 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,608 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:21,609 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,609 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:21,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:21,610 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,610 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:21,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:21,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:21,611 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:21,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:21,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38716 deadline: 1682042181611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:21,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:21,620 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:21,620 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:21,620 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:21,628 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:21,628 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:21,628 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:21,628 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:21,628 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:21,628 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:21,628 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:21,628 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:21,629 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,632 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:21,632 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:21,634 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:21,635 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:21,635 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,635 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:21,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:21,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1669884a[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:21,637 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:21,637 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,639 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,639 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:21,639 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:21,639 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:21,640 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,640 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:21,641 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:21,641 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,641 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:21,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:21,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:21,641 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:21,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:21,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:21,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:21,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:21,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:21,644 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:21,644 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:21,644 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:21,644 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:21,644 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:21,646 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,646 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:21,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,646 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:21,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:21,646 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,648 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,650 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:21,650 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:21,650 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,650 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:21,650 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:21,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:21,650 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:21,650 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:21,650 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:21,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:21,650 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:21,650 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,651 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,651 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,651 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,650 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:21,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:21,751 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:21,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:21,754 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:21,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:21,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:21,760 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:21,760 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:21,760 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:21,760 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:21,761 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,761 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:21,762 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:21,762 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:21,763 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,763 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:21,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:21,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:21,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:21,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:21,768 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:21,768 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:21,768 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:21,776 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:21,776 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:21,776 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:21,776 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,777 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:21,777 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:21,778 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,778 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:21,779 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,779 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:21,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:21,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@caf9afa[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:21,780 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:21,780 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,783 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,783 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:21,784 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,784 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:21,784 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:21,784 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:21,786 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,786 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:21,786 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:21,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:21,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:21,786 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:21,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:21,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:21,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:21,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:21,789 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:21,789 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:21,789 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:21,789 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:21,789 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:21,790 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:21,790 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:21,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,791 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:21,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:21,791 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:21,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,793 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:21,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682042181794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:21,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682042181795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:21,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:21,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38730 deadline: 1682042181795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38700 deadline: 1682042181795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,796 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:21,796 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:21,796 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:21,796 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:21,796 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:21,796 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:21,797 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:21,797 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,797 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,797 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:21,797 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:21,796 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:21,797 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:21,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:21,797 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:21,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38716 deadline: 1682042181816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,897 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:21,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:21,899 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:21,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:21,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:21,901 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:21,901 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:21,901 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:21,901 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:21,901 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,901 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:21,903 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:21,903 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:21,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:21,904 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,904 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:21,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:21,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:21,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:21,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:21,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:21,905 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:21,905 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:21,905 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:21,905 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:21,905 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:21,905 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:21,905 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:21,905 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:21,905 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:21,905 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:21,905 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:21,905 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:21,905 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,905 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:21,905 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:21,907 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,907 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:21,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:21,907 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,907 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:21,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:21,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:21,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:21,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:21,909 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:21,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14f2f541[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:21,909 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,910 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,910 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:21,910 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:21,910 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:21,910 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,911 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:21,914 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:21,914 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:21,914 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,914 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:21,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:21,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:21,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:21,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:21,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:21,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:21,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:21,917 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:21,917 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:21,917 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:21,917 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:21,917 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:21,918 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,918 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:21,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,918 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:21,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:21,919 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,920 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,922 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,922 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:21,922 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:21,922 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:21,922 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:21,922 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:21,922 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,922 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:21,922 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:21,922 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:21,922 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:21,922 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:21,922 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:21,922 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:21,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:21,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:21,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:21,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:21,926 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042130cb26c1241044a4a69a0134296fb224 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042130cb26c1241044a4a69a0134296fb224 2023-04-21 01:55:21,930 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/711650f3b39d464dbef6159ea38056f7 2023-04-21 01:55:21,938 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/7d45932cf5f84010be3a9e47ac43a292 2023-04-21 01:55:22,022 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:22,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:22,025 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:22,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:22,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:22,026 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:22,026 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:22,026 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:22,026 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:22,027 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,027 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:22,029 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:22,029 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:22,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:22,029 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,029 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:22,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:22,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:22,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:22,030 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:22,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:22,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:22,030 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:22,030 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:22,030 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:22,035 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:22,035 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:22,035 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:22,035 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:22,035 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:22,035 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:22,035 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:22,035 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:22,035 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,035 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:22,035 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:22,036 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,036 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:22,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:22,037 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,037 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:22,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:22,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:22,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:22,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:22,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:22,038 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:22,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@583602bb[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:22,038 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,039 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,039 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:22,039 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,039 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:22,039 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:22,039 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:22,040 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:22,040 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:22,040 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,040 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:22,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:22,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:22,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:22,043 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:22,044 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:22,044 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:22,044 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:22,044 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:22,046 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,046 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,046 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:22,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:22,046 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:22,047 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,049 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,049 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:22,049 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:22,049 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:22,049 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:22,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:22,049 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:22,049 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:22,050 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:22,050 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:22,049 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:22,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:22,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:22,050 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,050 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,050 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,050 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:22,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38730 deadline: 1682042182096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,105 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:22,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682042182098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,106 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:22,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682042182098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,106 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:22,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38700 deadline: 1682042182098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:22,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38716 deadline: 1682042182117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,150 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:22,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:22,152 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:22,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:22,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:22,153 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:22,153 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:22,153 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:22,153 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:22,153 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,153 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:22,154 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,154 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:22,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:22,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:22,155 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,155 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:22,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:22,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:22,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:22,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:22,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:22,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:22,156 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:22,156 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:22,156 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:22,156 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:22,156 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:22,156 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:22,156 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:22,156 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:22,156 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:22,156 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:22,156 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:22,156 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,156 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:22,156 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:22,157 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,157 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:22,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:22,158 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,158 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:22,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:22,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:22,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:22,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:22,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:22,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c7ea2f[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:22,159 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:22,159 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,161 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,161 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:22,161 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:22,161 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:22,161 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,162 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:22,163 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,163 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:22,163 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:22,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:22,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:22,163 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:22,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:22,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:22,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:22,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:22,165 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:22,165 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:22,165 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:22,165 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:22,165 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:22,166 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,166 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,166 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:22,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:22,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:22,167 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:22,168 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,171 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,171 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:22,171 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:22,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:22,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:22,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:22,171 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:22,171 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:22,171 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:22,171 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:22,172 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:22,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:22,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:22,172 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,172 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:22,172 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:22,172 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,172 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,172 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,272 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:22,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:22,275 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:22,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:22,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:22,276 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:22,276 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:22,276 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:22,276 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:22,277 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,277 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:22,279 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,279 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:22,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:22,279 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,279 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:22,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:22,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:22,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:22,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:22,281 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:22,281 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:22,281 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:22,281 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:22,281 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:22,281 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:22,281 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:22,281 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:22,281 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:22,281 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:22,281 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:22,281 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,281 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:22,281 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:22,283 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,283 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:22,283 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,283 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:22,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:22,285 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:22,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72cfd8c7[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:22,285 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,286 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,286 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:22,286 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:22,286 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:22,286 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,286 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:22,287 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:22,287 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:22,287 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,287 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:22,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:22,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:22,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:22,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:22,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:22,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:22,290 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:22,290 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:22,290 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:22,290 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:22,290 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:22,294 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,294 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,294 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:22,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:22,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:22,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,295 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:22,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:22,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:22,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:22,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:22,296 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:22,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:22,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:22,300 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,300 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:22,300 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:22,300 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,300 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,300 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:22,300 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:22,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:22,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:22,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:22,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:22,302 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:22,302 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,302 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:22,302 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:22,302 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:22,302 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:22,302 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:22,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:22,347 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/1c3d9293f6884bf4b3dafd86afddc580 2023-04-21 01:55:22,351 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/711650f3b39d464dbef6159ea38056f7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/711650f3b39d464dbef6159ea38056f7 2023-04-21 01:55:22,354 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/711650f3b39d464dbef6159ea38056f7, entries=150, sequenceid=17, filesize=25.1 K 2023-04-21 01:55:22,354 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/7d45932cf5f84010be3a9e47ac43a292 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/7d45932cf5f84010be3a9e47ac43a292 2023-04-21 01:55:22,358 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/7d45932cf5f84010be3a9e47ac43a292, entries=150, sequenceid=17, filesize=11.6 K 2023-04-21 01:55:22,359 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/1c3d9293f6884bf4b3dafd86afddc580 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/1c3d9293f6884bf4b3dafd86afddc580 2023-04-21 01:55:22,364 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/1c3d9293f6884bf4b3dafd86afddc580, entries=150, sequenceid=17, filesize=11.6 K 2023-04-21 01:55:22,365 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 810791807ce3c9f7e9f14187121bf792 in 905ms, sequenceid=17, compaction requested=false 2023-04-21 01:55:22,365 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:22,403 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:22,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:22,405 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:22,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:22,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:22,406 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:22,406 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:22,406 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:22,406 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:22,407 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,407 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:22,408 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:22,408 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:22,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:22,409 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,409 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:22,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:22,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:22,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:22,410 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:22,410 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:22,410 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:22,410 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:22,410 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:22,410 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:22,410 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:22,410 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 810791807ce3c9f7e9f14187121bf792 3/3 column families, dataSize=114.05 KB heapSize=325.22 KB 2023-04-21 01:55:22,410 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=A 2023-04-21 01:55:22,410 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:22,410 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=B 2023-04-21 01:55:22,410 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:22,410 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=C 2023-04-21 01:55:22,410 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:22,425 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421f1f3ab91011642cbb33439e8dd800ff5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421f1f3ab91011642cbb33439e8dd800ff5 2023-04-21 01:55:22,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:22,609 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:22,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:22,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38700 deadline: 1682042182624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:22,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38730 deadline: 1682042182624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:22,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682042182625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:22,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682042182625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:22,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38716 deadline: 1682042182626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:22,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38700 deadline: 1682042182732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:22,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38730 deadline: 1682042182732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:22,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682042182732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:22,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682042182732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,831 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/ca0087d054e5461585e3ecdb3ee62066 2023-04-21 01:55:22,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:22,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682042182935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:22,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682042182935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:22,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38700 deadline: 1682042182936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:22,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:22,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38730 deadline: 1682042182936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,109 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-21 01:55:23,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:23,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682042183239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:23,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38700 deadline: 1682042183240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:23,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682042183240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:23,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38730 deadline: 1682042183242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,247 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/7b5d376f693e4f7291382b161d7b44ca 2023-04-21 01:55:23,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:23,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38716 deadline: 1682042183641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,658 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/fac8d20d26714c0faea8821df7c92c19 2023-04-21 01:55:23,662 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/ca0087d054e5461585e3ecdb3ee62066 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/ca0087d054e5461585e3ecdb3ee62066 2023-04-21 01:55:23,666 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/ca0087d054e5461585e3ecdb3ee62066, entries=150, sequenceid=51, filesize=25.1 K 2023-04-21 01:55:23,667 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/7b5d376f693e4f7291382b161d7b44ca as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/7b5d376f693e4f7291382b161d7b44ca 2023-04-21 01:55:23,672 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/7b5d376f693e4f7291382b161d7b44ca, entries=150, sequenceid=51, filesize=11.6 K 2023-04-21 01:55:23,673 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/fac8d20d26714c0faea8821df7c92c19 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/fac8d20d26714c0faea8821df7c92c19 2023-04-21 01:55:23,678 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/fac8d20d26714c0faea8821df7c92c19, entries=150, sequenceid=51, filesize=11.6 K 2023-04-21 01:55:23,679 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~325.17 KB/332976, currentSize=67.09 KB/68700 for 810791807ce3c9f7e9f14187121bf792 in 1269ms, sequenceid=51, compaction requested=false 2023-04-21 01:55:23,679 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:23,679 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:23,679 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:23,679 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:23,679 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,679 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:23,679 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:23,682 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,682 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:23,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:23,682 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,682 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:23,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:23,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:23,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:23,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:23,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:23,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71947322[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:23,684 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:23,684 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,685 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,685 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:23,685 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:23,685 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:23,686 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,686 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:23,686 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,687 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:23,687 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:23,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:23,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:23,687 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:23,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:23,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:23,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:23,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:23,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:23,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:23,690 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:23,690 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:23,690 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:23,690 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:23,690 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:23,692 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,692 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,692 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:23,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:23,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:23,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:23,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:23,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:23,693 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:23,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:23,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:23,695 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:23,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:23,699 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:23,699 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:23,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:23,699 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:23,699 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:23,699 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:23,699 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:23,699 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,699 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:23,699 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:23,699 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:23,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:23,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:23,699 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,700 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,700 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,700 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:23,749 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 810791807ce3c9f7e9f14187121bf792 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-21 01:55:23,750 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=A 2023-04-21 01:55:23,750 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:23,750 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=B 2023-04-21 01:55:23,750 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:23,750 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=C 2023-04-21 01:55:23,750 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:23,767 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304216ad32ac92d8040a88a47572b19aab073 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304216ad32ac92d8040a88a47572b19aab073 2023-04-21 01:55:23,775 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=66, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/5822b1bd8aa048149b42825e8dd661b3 2023-04-21 01:55:23,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:23,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682042183770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:23,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38730 deadline: 1682042183771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:23,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682042183772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:23,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38700 deadline: 1682042183777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,799 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:23,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:23,801 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:23,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:23,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:23,803 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:23,803 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:23,803 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:23,803 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:23,803 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,803 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:23,804 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:23,804 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:23,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:23,805 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,805 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:23,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:23,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:23,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:23,806 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:23,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:23,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:23,806 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:23,806 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:23,806 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:23,807 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:23,807 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:23,807 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:23,807 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:23,808 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:23,808 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:23,808 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:23,808 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:23,808 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,808 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:23,808 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:23,809 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,809 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:23,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:23,809 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,809 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:23,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:23,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:23,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:23,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:23,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:23,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19df8471[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:23,811 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:23,811 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,812 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:23,812 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,812 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:23,814 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:23,814 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:23,814 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,814 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:23,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:23,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:23,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:23,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:23,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:23,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:23,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:23,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:23,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:23,817 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:23,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:23,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:23,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:23,817 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:23,819 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,819 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,819 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:23,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:23,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:23,819 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:23,820 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:23,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:23,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,823 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,823 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:23,823 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:23,823 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,823 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,823 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,823 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:23,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:23,823 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:23,823 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:23,823 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:23,823 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,823 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:23,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:23,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:23,823 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:23,823 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:23,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:23,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:23,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:23,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682042183880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:23,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:23,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38730 deadline: 1682042183881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682042183881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:23,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38700 deadline: 1682042183884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,923 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:23,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:23,926 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:23,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:23,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:23,927 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:23,927 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:23,927 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:23,927 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:23,928 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,928 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:23,929 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:23,929 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:23,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:23,929 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,929 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:23,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:23,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:23,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:23,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:23,930 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:23,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:23,930 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:23,930 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:23,930 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:23,930 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:23,930 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:23,930 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:23,930 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:23,931 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:23,931 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:23,931 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:23,931 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:23,931 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,931 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:23,931 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:23,934 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,934 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:23,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:23,935 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,935 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:23,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:23,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:23,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:23,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:23,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:23,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@247483f7[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:23,937 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:23,937 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,938 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,938 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:23,938 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,938 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:23,939 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:23,939 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:23,940 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:23,940 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:23,940 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,940 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:23,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:23,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:23,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:23,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:23,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:23,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:23,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:23,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:23,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:23,943 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:23,943 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:23,943 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:23,944 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:23,944 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:23,945 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,945 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:23,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,945 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:23,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:23,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:23,946 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:23,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:23,948 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:23,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:23,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,951 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:23,951 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:23,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:23,951 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:23,951 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:23,951 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:23,951 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:23,951 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:23,951 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:23,951 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:23,951 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:23,951 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:23,951 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:23,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:23,952 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:23,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:23,952 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,052 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:24,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:24,054 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:24,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:24,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:24,055 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:24,055 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:24,055 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:24,055 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:24,055 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,055 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,057 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,057 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:24,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:24,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:24,057 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,057 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:24,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:24,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,058 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:24,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:24,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:24,058 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:24,058 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:24,058 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:24,060 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:24,060 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:24,060 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:24,060 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:24,060 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:24,060 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:24,060 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:24,060 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:24,060 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,060 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:24,060 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:24,062 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,062 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:24,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:24,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:24,062 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,062 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:24,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:24,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:24,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:24,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28716e5e[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:24,064 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:24,064 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,066 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,067 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:24,067 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:24,067 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,067 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:24,067 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:24,071 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,071 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:24,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:24,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:24,071 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:24,071 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:24,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:24,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:24,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:24,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:24,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:24,074 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:24,074 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:24,074 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:24,074 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:24,074 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:24,075 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,075 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,075 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:24,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:24,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:24,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:24,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:24,076 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:24,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:24,078 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:24,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,081 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:24,081 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:24,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:24,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:24,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:24,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:24,082 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,082 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,082 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,082 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,082 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,082 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,082 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,082 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:24,082 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:24,082 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:24,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:24,082 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:24,082 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:24,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:24,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682042184088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:24,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682042184088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:24,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38730 deadline: 1682042184089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:24,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38700 deadline: 1682042184089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,183 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:24,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:24,185 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:24,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:24,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:24,186 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:24,187 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:24,187 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:24,187 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:24,187 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,187 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,188 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:24,188 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:24,189 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,189 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:24,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:24,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:24,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:24,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:24,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:24,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:24,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:24,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:24,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:24,190 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:24,190 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:24,190 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:24,190 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:24,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:24,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:24,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:24,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:24,192 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/0fe479e55fac449d8c3b3cbb8ffc833d 2023-04-21 01:55:24,192 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,193 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:24,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:24,193 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,193 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:24,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:24,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:24,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:24,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:24,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7aae0511[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:24,195 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:24,195 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,202 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,202 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:24,202 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:24,202 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:24,202 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,203 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:24,204 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:24,204 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,204 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:24,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:24,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:24,204 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:24,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:24,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:24,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:24,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:24,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:24,206 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:24,206 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:24,206 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:24,206 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:24,206 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:24,207 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,208 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:24,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:24,208 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:24,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:24,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:24,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:24,209 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,210 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:24,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,213 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,213 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:24,213 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:24,213 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,213 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:24,214 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:24,213 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:24,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:24,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:24,213 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,214 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:24,214 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:24,214 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,214 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:24,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:24,214 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,214 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,314 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:24,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:24,316 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:24,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:24,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:24,318 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:24,318 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:24,318 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:24,318 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:24,318 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,318 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,320 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,320 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:24,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:24,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:24,321 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,321 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:24,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:24,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:24,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:24,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:24,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:24,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:24,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:24,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:24,323 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:24,323 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:24,323 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:24,323 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:24,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:24,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:24,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:24,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:24,324 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,324 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:24,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:24,325 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:24,325 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:24,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:24,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:24,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:24,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fb7ea39[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:24,327 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:24,327 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,328 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,329 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:24,329 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:24,329 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:24,329 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,329 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:24,332 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:24,332 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:24,333 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:24,333 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:24,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:24,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:24,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:24,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:24,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:24,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:24,335 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:24,335 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:24,335 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:24,335 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:24,335 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:24,336 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,336 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:24,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:24,337 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:24,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:24,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:24,337 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:24,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:24,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,338 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:24,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,340 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,340 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,340 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:24,340 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:24,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:24,340 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:24,341 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:24,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:24,340 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:24,340 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,341 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,341 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,341 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,341 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,341 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:24,341 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:24,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:24,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:24,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:24,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:24,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38730 deadline: 1682042184391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:24,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38700 deadline: 1682042184393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:24,401 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:24,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682042184393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682042184393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,441 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:24,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:24,444 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:24,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:24,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:24,446 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:24,446 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:24,446 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:24,446 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:24,446 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,446 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,450 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:24,450 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:24,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:24,451 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,451 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:24,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:24,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,452 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:24,452 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:24,452 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:24,452 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:24,452 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:24,452 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:24,452 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:24,452 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:24,452 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:24,452 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:24,452 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:24,452 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:24,452 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:24,452 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:24,452 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,453 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:24,453 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:24,454 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,454 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:24,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:24,454 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,454 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:24,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:24,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:24,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f5cb903[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:24,456 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:24,456 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,457 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,457 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:24,457 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:24,457 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:24,458 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:24,458 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:24,458 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:24,458 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,459 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:24,459 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:24,459 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:24,459 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:24,459 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:24,459 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:24,459 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:24,461 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,462 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:24,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:24,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,462 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:24,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:24,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:24,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:24,462 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:24,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,463 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,465 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:24,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:24,465 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:24,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:24,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:24,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,465 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:24,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:24,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:24,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,466 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:24,466 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:24,566 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:24,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:24,569 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:24,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:24,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:24,571 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:24,571 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:24,571 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:24,571 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:24,571 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,571 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,573 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,573 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:24,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:24,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:24,573 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,573 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:24,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:24,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:24,574 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:24,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:24,574 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:24,574 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:24,574 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:24,574 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:24,574 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:24,574 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:24,574 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:24,575 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:24,575 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:24,575 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:24,575 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:24,575 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,575 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:24,575 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:24,576 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,576 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:24,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:24,576 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,576 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:24,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:24,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:24,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:24,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:24,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cfc3728[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:24,578 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:24,579 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,580 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,580 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:24,581 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:24,581 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:24,581 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,581 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:24,582 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:24,582 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:24,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,582 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:24,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:24,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:24,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:24,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:24,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:24,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:24,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:24,585 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:24,585 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:24,585 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:24,585 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:24,585 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:24,586 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,586 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:24,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:24,586 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:24,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:24,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:24,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:24,588 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:24,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,591 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:24,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:24,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,594 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,594 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,594 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:24,594 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:24,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:24,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:24,594 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:24,594 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,594 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:24,594 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:24,594 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,594 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,594 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:24,594 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,595 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:24,595 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:24,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:24,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:24,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:24,610 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/06d3668f656e4689a8a5b5bede6b2a27 2023-04-21 01:55:24,615 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/5822b1bd8aa048149b42825e8dd661b3 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/5822b1bd8aa048149b42825e8dd661b3 2023-04-21 01:55:24,619 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/5822b1bd8aa048149b42825e8dd661b3, entries=250, sequenceid=66, filesize=38.7 K 2023-04-21 01:55:24,622 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/0fe479e55fac449d8c3b3cbb8ffc833d as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/0fe479e55fac449d8c3b3cbb8ffc833d 2023-04-21 01:55:24,629 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/0fe479e55fac449d8c3b3cbb8ffc833d, entries=150, sequenceid=66, filesize=11.6 K 2023-04-21 01:55:24,630 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/06d3668f656e4689a8a5b5bede6b2a27 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/06d3668f656e4689a8a5b5bede6b2a27 2023-04-21 01:55:24,634 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/06d3668f656e4689a8a5b5bede6b2a27, entries=150, sequenceid=66, filesize=11.6 K 2023-04-21 01:55:24,634 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 810791807ce3c9f7e9f14187121bf792 in 885ms, sequenceid=66, compaction requested=true 2023-04-21 01:55:24,634 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:24,635 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:24,635 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:24,635 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:24,635 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:24,635 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:24,636 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91122 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:24,636 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:24,636 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 810791807ce3c9f7e9f14187121bf792/A is initiating minor compaction (all files) 2023-04-21 01:55:24,636 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 810791807ce3c9f7e9f14187121bf792/B is initiating minor compaction (all files) 2023-04-21 01:55:24,636 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 810791807ce3c9f7e9f14187121bf792/A in TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:24,636 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 810791807ce3c9f7e9f14187121bf792/B in TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:24,636 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/711650f3b39d464dbef6159ea38056f7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/ca0087d054e5461585e3ecdb3ee62066, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/5822b1bd8aa048149b42825e8dd661b3] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp, totalSize=89.0 K 2023-04-21 01:55:24,636 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/7d45932cf5f84010be3a9e47ac43a292, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/7b5d376f693e4f7291382b161d7b44ca, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/0fe479e55fac449d8c3b3cbb8ffc833d] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp, totalSize=34.8 K 2023-04-21 01:55:24,636 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 711650f3b39d464dbef6159ea38056f7, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1682042121430 2023-04-21 01:55:24,636 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 7d45932cf5f84010be3a9e47ac43a292, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1682042121430 2023-04-21 01:55:24,637 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting ca0087d054e5461585e3ecdb3ee62066, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682042121505 2023-04-21 01:55:24,637 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 7b5d376f693e4f7291382b161d7b44ca, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682042121505 2023-04-21 01:55:24,637 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 5822b1bd8aa048149b42825e8dd661b3, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1682042122621 2023-04-21 01:55:24,637 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 0fe479e55fac449d8c3b3cbb8ffc833d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1682042122621 2023-04-21 01:55:24,644 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 810791807ce3c9f7e9f14187121bf792#B#compaction#343 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:24,651 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 810791807ce3c9f7e9f14187121bf792#A#compaction#342 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:24,669 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/8385b6720d924881a800ee7ec842b097 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/8385b6720d924881a800ee7ec842b097 2023-04-21 01:55:24,672 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 810791807ce3c9f7e9f14187121bf792/B of 810791807ce3c9f7e9f14187121bf792 into 8385b6720d924881a800ee7ec842b097(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:24,672 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:24,672 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., storeName=810791807ce3c9f7e9f14187121bf792/B, priority=13, startTime=1682042124635; duration=0sec 2023-04-21 01:55:24,672 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:24,672 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:24,673 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:24,673 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 810791807ce3c9f7e9f14187121bf792/C is initiating minor compaction (all files) 2023-04-21 01:55:24,673 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 810791807ce3c9f7e9f14187121bf792/C in TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:24,674 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/1c3d9293f6884bf4b3dafd86afddc580, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/fac8d20d26714c0faea8821df7c92c19, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/06d3668f656e4689a8a5b5bede6b2a27] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp, totalSize=34.8 K 2023-04-21 01:55:24,674 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 1c3d9293f6884bf4b3dafd86afddc580, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1682042121430 2023-04-21 01:55:24,674 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting fac8d20d26714c0faea8821df7c92c19, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682042121505 2023-04-21 01:55:24,674 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 06d3668f656e4689a8a5b5bede6b2a27, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1682042122621 2023-04-21 01:55:24,679 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 810791807ce3c9f7e9f14187121bf792#C#compaction#344 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:24,695 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:24,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:24,697 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:24,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:24,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:24,699 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:24,699 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:24,699 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:24,699 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:24,699 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,699 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,701 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:24,701 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:24,702 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,702 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:24,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:24,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:24,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:24,709 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:24,709 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:24,709 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:24,709 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:24,709 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:24,709 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:24,709 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:24,710 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 810791807ce3c9f7e9f14187121bf792 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-21 01:55:24,710 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=A 2023-04-21 01:55:24,710 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:24,710 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=B 2023-04-21 01:55:24,710 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:24,710 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=C 2023-04-21 01:55:24,710 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:24,723 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421fd1df3e711574ab1845bb90842f1cc8d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421fd1df3e711574ab1845bb90842f1cc8d 2023-04-21 01:55:24,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:24,899 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:24,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:24,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38700 deadline: 1682042184914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:24,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682042184916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:24,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682042184917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:24,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:24,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38730 deadline: 1682042184919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:25,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38700 deadline: 1682042185020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:25,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682042185024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:25,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682042185024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:25,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38730 deadline: 1682042185024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,087 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/ebba7854f9684b5b90ac5837dd8d67ec as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/ebba7854f9684b5b90ac5837dd8d67ec 2023-04-21 01:55:25,091 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 810791807ce3c9f7e9f14187121bf792/C of 810791807ce3c9f7e9f14187121bf792 into ebba7854f9684b5b90ac5837dd8d67ec(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:25,091 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:25,091 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., storeName=810791807ce3c9f7e9f14187121bf792/C, priority=13, startTime=1682042124635; duration=0sec 2023-04-21 01:55:25,091 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:25,130 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=84, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/6ea033cb25d64750ac9f63bd6e7e45dc 2023-04-21 01:55:25,141 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/eae686432a0241a98c4ab77abd1ed998 2023-04-21 01:55:25,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:25,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38700 deadline: 1682042185225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:25,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682042185228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:25,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682042185228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:25,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38730 deadline: 1682042185228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:25,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38700 deadline: 1682042185531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:25,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682042185533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:25,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682042185535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:25,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38730 deadline: 1682042185535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,551 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/c02ff544d6c44698b8d7ed87a6c18bc9 2023-04-21 01:55:25,556 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/6ea033cb25d64750ac9f63bd6e7e45dc as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/6ea033cb25d64750ac9f63bd6e7e45dc 2023-04-21 01:55:25,568 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/6ea033cb25d64750ac9f63bd6e7e45dc, entries=150, sequenceid=84, filesize=25.1 K 2023-04-21 01:55:25,569 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/eae686432a0241a98c4ab77abd1ed998 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/eae686432a0241a98c4ab77abd1ed998 2023-04-21 01:55:25,573 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/eae686432a0241a98c4ab77abd1ed998, entries=150, sequenceid=84, filesize=11.6 K 2023-04-21 01:55:25,574 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/c02ff544d6c44698b8d7ed87a6c18bc9 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/c02ff544d6c44698b8d7ed87a6c18bc9 2023-04-21 01:55:25,580 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/c02ff544d6c44698b8d7ed87a6c18bc9, entries=150, sequenceid=84, filesize=11.6 K 2023-04-21 01:55:25,581 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 810791807ce3c9f7e9f14187121bf792 in 872ms, sequenceid=84, compaction requested=false 2023-04-21 01:55:25,581 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:25,581 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:25,581 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:25,581 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:25,581 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,584 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:25,584 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:25,585 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,585 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:25,586 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,586 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:25,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:25,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36f03b2b[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:25,587 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:25,587 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,588 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,588 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:25,588 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:25,588 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:25,589 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:25,590 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,590 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:25,590 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:25,590 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:25,590 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:25,590 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:25,590 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:25,590 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:25,590 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:25,590 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:25,593 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,593 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,593 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:25,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:25,593 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:25,594 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,597 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,597 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:25,597 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:25,597 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,597 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:25,597 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:25,597 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,597 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,597 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:25,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:25,597 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:25,597 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:25,597 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:25,598 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:25,598 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:25,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:25,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:25,655 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 810791807ce3c9f7e9f14187121bf792 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-21 01:55:25,655 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=A 2023-04-21 01:55:25,655 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:25,655 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=B 2023-04-21 01:55:25,655 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:25,655 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=C 2023-04-21 01:55:25,655 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:25,698 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:25,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:25,699 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:25,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:25,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:25,701 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:25,701 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:25,701 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:25,701 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:25,701 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,701 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:25,702 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,702 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:25,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:25,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:25,703 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,703 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:25,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:25,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:25,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:25,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:25,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:25,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:25,704 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:25,704 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:25,704 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:25,704 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:25,704 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:25,705 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:25,705 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:25,705 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:25,705 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:25,705 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:25,705 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:25,705 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,705 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:25,705 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:25,711 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,711 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:25,712 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,712 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:25,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:25,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16b6e9ee[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:25,713 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:25,713 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,714 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,714 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:25,714 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,714 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:25,714 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:25,714 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:25,715 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:25,715 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,715 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:25,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,715 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:25,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:25,717 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:25,717 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:25,717 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:25,717 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:25,717 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:25,719 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,719 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:25,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,719 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:25,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:25,720 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:25,721 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,724 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,724 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:25,724 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:25,724 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:25,725 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:25,724 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:25,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:25,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:25,724 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:25,724 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:25,725 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:25,725 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,725 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:25,725 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:25,725 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,725 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:25,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:25,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38716 deadline: 1682042185749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,825 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:25,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:25,827 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:25,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:25,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:25,832 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:25,832 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:25,832 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:25,832 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:25,833 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,833 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:25,834 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:25,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:25,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:25,834 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,834 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:25,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:25,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:25,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:25,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:25,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:25,835 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:25,835 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:25,835 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:25,836 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:25,836 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:25,836 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:25,836 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:25,836 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:25,836 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:25,836 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:25,836 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:25,836 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,836 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:25,836 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:25,838 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,839 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,839 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:25,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:25,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:25,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:25,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:25,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c58de9a[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:25,841 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:25,841 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,842 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,842 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:25,842 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:25,842 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,842 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:25,842 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:25,845 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:25,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,848 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:25,848 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:25,848 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:25,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:25,848 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:25,848 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:25,848 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:25,848 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:25,848 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:25,850 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,850 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,850 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:25,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:25,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:25,850 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:25,852 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:25,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:25,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:25,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,856 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:25,856 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:25,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:25,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:25,856 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,856 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:25,856 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:25,856 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:25,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:25,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:25,856 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:25,856 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:25,856 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:25,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:25,856 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:25,856 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,856 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,856 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,856 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:25,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38716 deadline: 1682042185857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,881 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/13aae40fa5b640309f9cc9570d01607b as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/13aae40fa5b640309f9cc9570d01607b 2023-04-21 01:55:25,885 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 810791807ce3c9f7e9f14187121bf792/A of 810791807ce3c9f7e9f14187121bf792 into 13aae40fa5b640309f9cc9570d01607b(size=25.2 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-21 01:55:25,885 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:25,885 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., storeName=810791807ce3c9f7e9f14187121bf792/A, priority=13, startTime=1682042124634; duration=1sec 2023-04-21 01:55:25,885 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:25,956 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:25,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:25,959 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:25,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:25,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:25,960 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:25,960 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:25,960 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:25,960 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:25,961 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,961 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:25,962 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:25,962 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:25,962 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,962 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:25,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:25,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:25,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:25,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:25,964 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:25,964 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:25,964 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:25,968 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:25,968 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:25,968 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:25,968 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:25,968 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:25,968 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:25,968 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:25,968 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:25,968 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,968 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:25,968 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:25,970 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,970 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:25,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:25,971 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,971 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:25,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:25,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:25,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:25,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:25,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:25,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31c0e14e[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:25,972 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:25,972 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,973 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,973 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:25,973 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:25,973 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:25,974 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,974 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:25,975 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,975 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:25,975 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:25,975 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:25,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:25,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:25,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:25,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:25,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:25,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:25,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:25,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:25,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:25,977 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:25,977 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:25,977 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:25,977 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:25,977 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:25,979 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,979 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,979 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:25,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:25,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:25,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:25,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:25,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:25,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:25,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:25,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:25,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:25,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:25,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,984 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,985 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,993 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,993 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:25,993 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:25,993 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:25,993 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,993 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:25,993 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:25,993 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:25,993 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:25,993 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:25,993 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:25,993 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:25,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:25,994 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:25,994 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:25,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:26,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:26,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38700 deadline: 1682042186036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:26,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682042186037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:26,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682042186041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:26,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38730 deadline: 1682042186047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:26,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38716 deadline: 1682042186061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,068 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304219715a252c32a4ed394108f7707fcea66 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304219715a252c32a4ed394108f7707fcea66 2023-04-21 01:55:26,094 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:26,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:26,096 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:26,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:26,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:26,099 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:26,099 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:26,099 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:26,099 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:26,099 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,099 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,101 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,101 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,101 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:26,101 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:26,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:26,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:26,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:26,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:26,103 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:26,103 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:26,103 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:26,103 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:26,103 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:26,103 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:26,103 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:26,103 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:26,103 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:26,104 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:26,104 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:26,104 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:26,104 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:26,104 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,104 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:26,104 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:26,106 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,106 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:26,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:26,106 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,106 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:26,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:26,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:26,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:26,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:26,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6debd3e[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:26,108 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:26,108 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,114 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,114 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:26,114 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:26,115 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:26,115 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,115 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:26,119 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,119 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:26,119 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:26,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,120 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:26,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:26,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:26,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:26,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:26,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:26,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:26,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:26,123 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:26,123 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:26,123 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:26,123 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:26,123 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:26,124 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,124 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:26,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,124 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:26,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:26,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:26,125 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:26,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:26,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,126 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:26,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,129 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,129 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:26,129 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:26,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:26,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:26,129 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,129 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:26,130 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:26,129 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:26,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:26,130 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,130 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:26,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:26,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:26,130 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:26,130 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,130 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,130 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,130 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,230 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:26,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:26,233 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:26,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:26,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:26,236 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:26,236 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:26,236 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:26,236 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:26,236 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,236 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,237 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:26,237 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:26,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:26,238 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,238 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:26,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:26,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:26,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:26,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:26,244 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:26,244 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:26,244 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:26,248 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:26,248 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:26,248 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:26,248 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:26,248 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:26,248 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:26,248 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:26,248 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:26,248 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,248 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:26,248 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:26,251 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,251 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:26,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:26,252 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:26,252 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:26,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:26,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2786cd92[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:26,253 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:26,253 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,254 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,254 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:26,254 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:26,254 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:26,254 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,254 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:26,256 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:26,256 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,256 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:26,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:26,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:26,256 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:26,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:26,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:26,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:26,258 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:26,258 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:26,258 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:26,258 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:26,258 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:26,259 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,259 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:26,259 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:26,260 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:26,261 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,265 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,265 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,265 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:26,265 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:26,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:26,265 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:26,265 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:26,265 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,265 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,265 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,265 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,265 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:26,265 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:26,265 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:26,265 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:26,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:26,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:26,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:26,366 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:26,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:26,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:26,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38716 deadline: 1682042186367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,372 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:26,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:26,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:26,376 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:26,376 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:26,376 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:26,376 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:26,376 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,376 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,377 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,377 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:26,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:26,378 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,378 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:26,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:26,383 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:26,383 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:26,383 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:26,383 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:26,384 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:26,384 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:26,385 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:26,385 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:26,385 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:26,385 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:26,385 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:26,385 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:26,385 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,386 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:26,386 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:26,387 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,387 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:26,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:26,388 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,388 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:26,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:26,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d806bf6[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:26,389 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:26,389 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,390 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,390 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:26,390 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:26,390 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:26,390 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,390 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:26,393 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,393 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:26,393 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:26,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:26,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:26,393 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:26,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:26,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:26,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:26,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:26,396 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:26,396 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:26,396 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:26,396 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:26,396 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:26,398 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,398 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,398 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:26,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:26,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:26,398 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:26,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:26,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:26,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:26,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:26,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,400 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,402 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,402 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,402 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,402 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:26,402 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:26,402 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,402 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:26,402 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:26,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:26,402 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,402 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:26,402 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,402 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:26,402 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:26,402 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:26,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:26,474 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=101, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/376e2d48db9548f8b5f5d1a7a2232e7a 2023-04-21 01:55:26,503 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:26,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:26,507 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:26,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:26,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:26,509 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:26,509 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:26,509 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:26,509 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:26,509 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,509 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,510 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:26,510 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:26,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:26,511 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,511 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:26,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:26,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,511 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:26,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:26,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:26,512 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:26,512 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:26,512 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:26,516 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:26,516 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:26,516 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:26,516 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:26,516 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:26,516 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:26,516 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:26,516 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:26,516 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,516 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:26,516 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:26,523 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,524 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,524 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:26,525 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:26,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:26,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:26,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:26,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:26,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:26,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bfc2199[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:26,527 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:26,527 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,529 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,529 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:26,530 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:26,530 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:26,531 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,531 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:26,532 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:26,532 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:26,532 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:26,532 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:26,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:26,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:26,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:26,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:26,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:26,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:26,534 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:26,535 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:26,535 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:26,535 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:26,535 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:26,539 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,540 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:26,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,540 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:26,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:26,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:26,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:26,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:26,540 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:26,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,542 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:26,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,544 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:26,544 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:26,544 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:26,544 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:26,544 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:26,544 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:26,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:26,544 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:26,544 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:26,544 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,544 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:26,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:26,544 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,545 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,545 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,545 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,644 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:26,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:26,647 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:26,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:26,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:26,648 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:26,648 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:26,648 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:26,648 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:26,648 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,648 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,649 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:26,649 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:26,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:26,650 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,650 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:26,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:26,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:26,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:26,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:26,651 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:26,651 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:26,651 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:26,651 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:26,651 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:26,651 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:26,651 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:26,651 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:26,651 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:26,651 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:26,651 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:26,651 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,651 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:26,651 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:26,653 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,653 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:26,653 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,653 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:26,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:26,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:26,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:26,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48b85c95[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:26,657 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:26,657 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,662 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,662 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:26,662 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:26,662 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:26,662 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,662 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:26,663 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,663 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:26,663 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:26,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:26,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:26,664 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:26,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:26,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:26,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:26,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:26,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:26,666 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:26,666 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:26,666 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:26,666 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:26,666 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:26,667 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,668 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:26,668 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:26,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:26,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:26,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:26,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:26,668 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:26,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,669 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:26,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,671 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,671 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,671 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,671 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:26,671 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:26,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:26,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:26,672 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:26,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:26,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:26,672 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:26,672 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:26,672 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:26,672 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:26,673 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,673 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,673 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,673 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:26,772 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:26,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:26,774 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:26,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:26,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:26,775 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:26,776 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:26,776 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:26,776 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:26,776 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,776 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,778 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,778 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:26,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:26,778 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,778 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:26,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:26,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:26,779 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:26,779 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:26,779 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:26,779 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:26,779 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:26,779 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:26,779 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:26,780 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:26,780 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:26,780 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:26,780 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:26,780 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,780 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:26,780 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:26,781 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,781 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:26,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:26,781 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,781 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:26,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:26,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:26,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:26,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:26,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5943e24d[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:26,782 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:26,782 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,783 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,784 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:26,784 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:26,784 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:26,784 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,784 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:26,785 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,785 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:26,785 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:26,785 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:26,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:26,787 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:26,787 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:26,787 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:26,787 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:26,787 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:26,790 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,790 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,790 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:26,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:26,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:26,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,790 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:26,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:26,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:26,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:26,791 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:26,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,793 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,793 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,793 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,793 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:26,793 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,793 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,793 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,793 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,793 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:26,794 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:26,794 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:26,794 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:26,794 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:26,794 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:26,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:26,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:26,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:26,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:26,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38716 deadline: 1682042186875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,894 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:26,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:26,896 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:26,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:26,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:26,898 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:26,898 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:26,898 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:26,898 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:26,898 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/c5a0465631e9403a806495fd5d40fb58 2023-04-21 01:55:26,898 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,898 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,900 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,900 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:26,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:26,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:26,900 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,900 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:26,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:26,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:26,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:26,901 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:26,901 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:26,901 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:26,902 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:26,902 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:26,902 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:26,903 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:26,903 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:26,903 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:26,903 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:26,903 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:26,903 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,903 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:26,903 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:26,904 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,904 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:26,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:26,905 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,905 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:26,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:26,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:26,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5712ed48[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:26,907 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:26,907 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,909 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,910 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:26,910 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:26,910 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,910 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:26,910 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:26,911 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:26,911 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:26,911 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:26,911 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:26,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:26,913 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:26,913 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:26,914 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:26,914 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:26,914 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:26,915 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,915 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,915 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:26,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:26,917 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,918 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,929 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:26,929 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,929 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,929 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:26,929 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:26,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:26,929 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:26,929 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:26,929 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:26,929 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:26,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:26,929 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,929 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:26,929 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:26,929 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:26,930 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:26,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:26,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:27,030 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:27,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:27,032 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:27,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:27,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:27,033 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:27,033 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:27,033 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:27,033 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:27,033 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,033 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:27,035 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:27,035 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:27,035 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,035 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:27,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:27,036 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:27,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:27,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:27,036 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:27,036 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:27,036 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:27,037 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:27,037 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:27,037 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:27,037 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:27,037 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:27,037 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:27,037 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:27,037 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:27,037 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,037 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:27,037 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:27,038 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,038 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:27,039 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,039 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:27,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:27,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d38eeda[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:27,040 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:27,040 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,047 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,048 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:27,048 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,048 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:27,048 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:27,048 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:27,049 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:27,049 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:27,049 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:27,049 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:27,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:27,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:27,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:27,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:27,051 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:27,051 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:27,051 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:27,051 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:27,051 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:27,052 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:27,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682042187047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,052 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:27,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38700 deadline: 1682042187049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,053 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,053 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:27,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:27,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:27,053 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,054 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:27,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:27,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:27,055 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:27,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,056 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:27,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682042187051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,057 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,057 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:27,057 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:27,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:27,057 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:27,057 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:27,058 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:27,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:27,057 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:27,058 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:27,058 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,058 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:27,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:27,058 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,058 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:27,058 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,058 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:27,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38730 deadline: 1682042187055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,158 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:27,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:27,162 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:27,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:27,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:27,164 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:27,164 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:27,164 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:27,164 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:27,164 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,164 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:27,167 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:27,167 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:27,167 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,167 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:27,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:27,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:27,172 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:27,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:27,172 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:27,172 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:27,172 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:27,174 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:27,175 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:27,175 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:27,175 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:27,175 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:27,175 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:27,175 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:27,175 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:27,175 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,175 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:27,175 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:27,177 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,177 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:27,178 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,178 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:27,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:27,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ee471c[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:27,181 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:27,181 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,182 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,182 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:27,182 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:27,182 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:27,182 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,182 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:27,184 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:27,184 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,184 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:27,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:27,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:27,184 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:27,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:27,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:27,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:27,192 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:27,192 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:27,192 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:27,192 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:27,192 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:27,196 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:27,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:27,196 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,196 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:27,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:27,197 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:27,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:27,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,198 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,201 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,201 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:27,201 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:27,201 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:27,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:27,201 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:27,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:27,201 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:27,202 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:27,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:27,201 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,201 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:27,202 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,202 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,202 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:27,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:27,202 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:27,202 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:27,302 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:27,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:27,304 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:27,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:27,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:27,308 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:27,308 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:27,308 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:27,308 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:27,309 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,309 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:27,309 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/cbc01504270c4a8ab34fd780c87b9354 2023-04-21 01:55:27,310 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,310 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:27,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:27,311 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,311 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:27,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:27,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:27,313 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/376e2d48db9548f8b5f5d1a7a2232e7a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/376e2d48db9548f8b5f5d1a7a2232e7a 2023-04-21 01:55:27,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:27,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:27,316 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:27,316 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:27,316 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:27,318 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/376e2d48db9548f8b5f5d1a7a2232e7a, entries=200, sequenceid=101, filesize=31.9 K 2023-04-21 01:55:27,319 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/c5a0465631e9403a806495fd5d40fb58 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/c5a0465631e9403a806495fd5d40fb58 2023-04-21 01:55:27,320 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:27,320 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:27,320 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:27,320 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:27,320 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:27,320 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:27,320 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:27,320 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:27,320 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,320 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:27,320 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:27,322 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:27,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:27,322 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,322 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,322 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:27,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:27,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:27,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41f98c26[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:27,325 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:27,325 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,327 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,327 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,327 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:27,327 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:27,327 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:27,327 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:27,329 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,329 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:27,329 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:27,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:27,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:27,329 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:27,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:27,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:27,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/c5a0465631e9403a806495fd5d40fb58, entries=150, sequenceid=101, filesize=11.6 K 2023-04-21 01:55:27,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:27,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/cbc01504270c4a8ab34fd780c87b9354 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/cbc01504270c4a8ab34fd780c87b9354 2023-04-21 01:55:27,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:27,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:27,332 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:27,332 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:27,332 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:27,332 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:27,332 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:27,334 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,334 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:27,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:27,334 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:27,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:27,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:27,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:27,335 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:27,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,336 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/cbc01504270c4a8ab34fd780c87b9354, entries=150, sequenceid=101, filesize=11.6 K 2023-04-21 01:55:27,336 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:27,337 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 810791807ce3c9f7e9f14187121bf792 in 1682ms, sequenceid=101, compaction requested=true 2023-04-21 01:55:27,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,337 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:27,337 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:27,337 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:27,337 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:27,337 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:27,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,337 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:27,338 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:27,338 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 810791807ce3c9f7e9f14187121bf792/B is initiating minor compaction (all files) 2023-04-21 01:55:27,338 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 810791807ce3c9f7e9f14187121bf792/B in TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:27,338 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/8385b6720d924881a800ee7ec842b097, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/eae686432a0241a98c4ab77abd1ed998, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/c5a0465631e9403a806495fd5d40fb58] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp, totalSize=34.9 K 2023-04-21 01:55:27,339 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84281 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:27,339 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 810791807ce3c9f7e9f14187121bf792/A is initiating minor compaction (all files) 2023-04-21 01:55:27,339 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 810791807ce3c9f7e9f14187121bf792/A in TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:27,339 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/13aae40fa5b640309f9cc9570d01607b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/6ea033cb25d64750ac9f63bd6e7e45dc, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/376e2d48db9548f8b5f5d1a7a2232e7a] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp, totalSize=82.3 K 2023-04-21 01:55:27,339 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 8385b6720d924881a800ee7ec842b097, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1682042122621 2023-04-21 01:55:27,339 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting eae686432a0241a98c4ab77abd1ed998, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1682042123765 2023-04-21 01:55:27,340 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 13aae40fa5b640309f9cc9570d01607b, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1682042122621 2023-04-21 01:55:27,340 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting c5a0465631e9403a806495fd5d40fb58, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1682042124915 2023-04-21 01:55:27,340 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 6ea033cb25d64750ac9f63bd6e7e45dc, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1682042123765 2023-04-21 01:55:27,340 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 376e2d48db9548f8b5f5d1a7a2232e7a, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1682042124910 2023-04-21 01:55:27,343 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,343 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:27,343 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:27,343 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,343 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:27,343 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:27,344 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:27,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:27,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:27,343 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:27,344 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:27,343 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,344 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,344 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:27,344 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:27,344 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:27,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:27,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:27,357 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 810791807ce3c9f7e9f14187121bf792#B#compaction#351 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:27,377 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 810791807ce3c9f7e9f14187121bf792#A#compaction#352 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:27,444 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:27,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:27,446 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:27,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:27,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:27,448 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:27,448 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:27,448 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:27,448 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:27,448 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,448 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:27,449 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,449 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:27,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:27,450 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,450 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:27,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:27,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:27,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:27,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:27,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:27,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:27,451 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:27,451 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:27,451 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:27,451 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:27,451 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:27,451 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:27,451 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:27,451 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 810791807ce3c9f7e9f14187121bf792 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-21 01:55:27,451 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=A 2023-04-21 01:55:27,451 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:27,451 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=B 2023-04-21 01:55:27,451 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:27,451 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=C 2023-04-21 01:55:27,451 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:27,466 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304219bd41941d2d74be18e3a75c3da2567e5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304219bd41941d2d74be18e3a75c3da2567e5 2023-04-21 01:55:27,779 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/7450abb6ed034cc0924c6a91ce9ae8a1 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/7450abb6ed034cc0924c6a91ce9ae8a1 2023-04-21 01:55:27,782 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 810791807ce3c9f7e9f14187121bf792/B of 810791807ce3c9f7e9f14187121bf792 into 7450abb6ed034cc0924c6a91ce9ae8a1(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:27,783 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:27,783 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., storeName=810791807ce3c9f7e9f14187121bf792/B, priority=13, startTime=1682042127337; duration=0sec 2023-04-21 01:55:27,783 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:27,783 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:27,784 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:27,784 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 810791807ce3c9f7e9f14187121bf792/C is initiating minor compaction (all files) 2023-04-21 01:55:27,784 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 810791807ce3c9f7e9f14187121bf792/C in TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:27,784 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/ebba7854f9684b5b90ac5837dd8d67ec, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/c02ff544d6c44698b8d7ed87a6c18bc9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/cbc01504270c4a8ab34fd780c87b9354] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp, totalSize=34.9 K 2023-04-21 01:55:27,784 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting ebba7854f9684b5b90ac5837dd8d67ec, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1682042122621 2023-04-21 01:55:27,784 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting c02ff544d6c44698b8d7ed87a6c18bc9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1682042123765 2023-04-21 01:55:27,785 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting cbc01504270c4a8ab34fd780c87b9354, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1682042124915 2023-04-21 01:55:27,790 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 810791807ce3c9f7e9f14187121bf792#C#compaction#354 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:27,798 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/45c5728448024b9b96ba0ebedd40abdf as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/45c5728448024b9b96ba0ebedd40abdf 2023-04-21 01:55:27,803 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 810791807ce3c9f7e9f14187121bf792/C of 810791807ce3c9f7e9f14187121bf792 into 45c5728448024b9b96ba0ebedd40abdf(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:27,804 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:27,804 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., storeName=810791807ce3c9f7e9f14187121bf792/C, priority=13, startTime=1682042127337; duration=0sec 2023-04-21 01:55:27,804 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:27,881 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=118, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/7d3cd9326f194cc79647a55b07418acf 2023-04-21 01:55:27,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:27,885 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:27,898 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/5f02dbae1d4e45d0859ba6e5dbf69b4d 2023-04-21 01:55:27,914 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/44f43da7d4ed48e1b6c3f2225ca7b036 2023-04-21 01:55:27,918 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/7d3cd9326f194cc79647a55b07418acf as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/7d3cd9326f194cc79647a55b07418acf 2023-04-21 01:55:27,922 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/7d3cd9326f194cc79647a55b07418acf, entries=150, sequenceid=118, filesize=25.1 K 2023-04-21 01:55:27,922 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/5f02dbae1d4e45d0859ba6e5dbf69b4d as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/5f02dbae1d4e45d0859ba6e5dbf69b4d 2023-04-21 01:55:27,925 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/5f02dbae1d4e45d0859ba6e5dbf69b4d, entries=150, sequenceid=118, filesize=11.6 K 2023-04-21 01:55:27,926 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-21 01:55:27,927 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/44f43da7d4ed48e1b6c3f2225ca7b036 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/44f43da7d4ed48e1b6c3f2225ca7b036 2023-04-21 01:55:27,930 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/44f43da7d4ed48e1b6c3f2225ca7b036, entries=150, sequenceid=118, filesize=11.6 K 2023-04-21 01:55:27,931 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=46.96 KB/48090 for 810791807ce3c9f7e9f14187121bf792 in 480ms, sequenceid=118, compaction requested=false 2023-04-21 01:55:27,931 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:27,931 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:27,931 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:27,931 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:27,931 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,931 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:27,931 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:27,933 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:27,933 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:27,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:27,934 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,934 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:27,934 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 810791807ce3c9f7e9f14187121bf792 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-21 01:55:27,934 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=A 2023-04-21 01:55:27,934 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:27,934 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=B 2023-04-21 01:55:27,934 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:27,934 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=C 2023-04-21 01:55:27,934 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:27,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:27,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c141fc[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:27,936 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:27,936 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,938 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,938 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:27,938 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:27,938 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:27,938 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,938 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:27,940 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:27,940 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:27,940 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:27,940 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:27,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:27,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:27,943 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:27,943 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:27,943 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:27,943 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:27,943 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:27,944 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,944 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,944 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:27,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:27,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:27,945 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,946 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,948 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:27,948 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,948 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:27,948 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:27,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:27,948 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:27,948 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:27,949 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:27,948 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:27,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:27,949 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:27,949 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:27,949 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:27,949 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,949 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:27,949 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:27,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:28,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042128024, max timestamp=1682042128037Num uniques 859; 2023-04-21 01:55:28,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:28,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042128024, max timestamp=1682042128037Num uniques 859; 2023-04-21 01:55:28,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042128024, max timestamp=1682042128037Num uniques 859; 2023-04-21 01:55:28,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:28,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:28,049 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:28,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:28,051 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:28,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:28,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:28,053 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:28,053 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:28,053 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:28,053 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:28,053 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,053 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,055 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:28,055 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:28,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:28,056 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,056 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:28,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:28,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:28,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:28,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:28,057 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:28,057 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:28,057 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:28,057 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:28,057 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:28,057 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:28,057 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:28,057 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:28,057 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:28,057 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:28,057 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:28,057 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,058 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:28,058 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:28,059 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,059 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:28,059 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,059 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:28,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:28,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:28,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b452120[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:28,061 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:28,061 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,062 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,062 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:28,062 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:28,063 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:28,062 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,063 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:28,064 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,064 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:28,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:28,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:28,064 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:28,064 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:28,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:28,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:28,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:28,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:28,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:28,066 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:28,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:28,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:28,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:28,066 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:28,068 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,068 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:28,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:28,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,069 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:28,069 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:28,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:28,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:28,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:28,071 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:28,073 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:28,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:28,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:28,073 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:28,073 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:28,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:28,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,074 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:28,074 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:28,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:28,074 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,074 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:28,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38716 deadline: 1682042188126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,174 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:28,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:28,177 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:28,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:28,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:28,178 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:28,178 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:28,178 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:28,178 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:28,184 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,184 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,185 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:28,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:28,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,186 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,186 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,186 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:28,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:28,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:28,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:28,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:28,187 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:28,187 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:28,187 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:28,187 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:28,187 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:28,187 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:28,188 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:28,188 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:28,188 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:28,188 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:28,188 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:28,188 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,188 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:28,188 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:28,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,190 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:28,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:28,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:28,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:28,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:28,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:28,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:28,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e6659f3[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:28,194 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:28,194 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,196 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,196 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:28,196 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,196 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:28,196 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:28,196 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:28,198 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:28,198 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:28,198 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:28,198 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:28,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:28,200 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:28,200 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:28,200 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:28,200 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:28,200 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:28,202 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,202 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,202 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:28,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:28,203 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:28,203 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,206 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:28,206 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:28,206 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:28,206 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:28,206 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:28,206 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:28,206 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,206 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,206 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,206 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,206 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,206 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,206 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:28,207 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:28,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:28,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:28,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:28,211 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/551df64958b94bb5b4d2a214ab89510e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/551df64958b94bb5b4d2a214ab89510e 2023-04-21 01:55:28,214 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 810791807ce3c9f7e9f14187121bf792/A of 810791807ce3c9f7e9f14187121bf792 into 551df64958b94bb5b4d2a214ab89510e(size=25.3 K), total size for store is 50.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:28,214 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:28,215 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., storeName=810791807ce3c9f7e9f14187121bf792/A, priority=13, startTime=1682042127337; duration=0sec 2023-04-21 01:55:28,215 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:28,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:28,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38716 deadline: 1682042188234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,307 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:28,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:28,309 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:28,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:28,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:28,310 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:28,310 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:28,310 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:28,310 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:28,311 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,311 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,312 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,312 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:28,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:28,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:28,313 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,313 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:28,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:28,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:28,314 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:28,314 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:28,314 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:28,314 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:28,314 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:28,314 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:28,315 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,315 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:28,315 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:28,319 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,319 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:28,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:28,319 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,319 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:28,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:28,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:28,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:28,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:28,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33f36aa5[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:28,320 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:28,320 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,321 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,322 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:28,322 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:28,322 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:28,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:28,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:28,323 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:28,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:28,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:28,325 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:28,325 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:28,325 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:28,325 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:28,325 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:28,327 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,327 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:28,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,327 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:28,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:28,328 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:28,329 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,331 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,331 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,331 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:28,331 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:28,331 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,331 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,331 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,331 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,331 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:28,331 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:28,331 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:28,332 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:28,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:28,332 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:28,332 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:28,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:28,347 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304211651fca4118343eabf1014a0081216d7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304211651fca4118343eabf1014a0081216d7 2023-04-21 01:55:28,432 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:28,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:28,434 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:28,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:28,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:28,436 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:28,436 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:28,436 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:28,436 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:28,436 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,436 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,437 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,438 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,438 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:28,439 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:28,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:28,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:28,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:28,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:28,445 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:28,445 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:28,445 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:28,448 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:28,448 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:28,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:28,448 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:28,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38716 deadline: 1682042188439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,448 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:28,448 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:28,448 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:28,448 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:28,448 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:28,449 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,449 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:28,449 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:28,451 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,451 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:28,451 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,451 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:28,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:28,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38b61386[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:28,453 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:28,453 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,455 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:28,455 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:28,455 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:28,456 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:28,456 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:28,456 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:28,457 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,457 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:28,457 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:28,457 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:28,457 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:28,457 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:28,457 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:28,457 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:28,458 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,458 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:28,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,458 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:28,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:28,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:28,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:28,459 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:28,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,461 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:28,466 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:28,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:28,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:28,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:28,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:28,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,466 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:28,466 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:28,467 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:28,467 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:28,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:28,567 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:28,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:28,569 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:28,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:28,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:28,571 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:28,571 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:28,571 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:28,571 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:28,571 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,571 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,573 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,573 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,573 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:28,574 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:28,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:28,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:28,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:28,579 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:28,579 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:28,579 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:28,579 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:28,579 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:28,579 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:28,579 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:28,579 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:28,579 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:28,579 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:28,579 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:28,580 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,580 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:28,580 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:28,581 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,581 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:28,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:28,581 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,582 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:28,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:28,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:28,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:28,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:28,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a987991[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:28,583 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:28,583 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,584 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,584 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:28,584 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:28,584 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:28,585 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,585 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:28,586 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,586 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:28,586 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:28,586 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:28,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:28,588 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:28,588 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:28,589 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:28,589 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:28,589 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:28,591 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,591 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,591 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:28,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:28,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:28,591 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:28,592 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,594 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:28,594 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:28,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:28,595 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,595 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,595 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,595 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:28,595 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,595 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:28,595 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:28,595 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,595 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,595 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:28,595 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:28,595 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:28,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:28,695 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:28,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:28,697 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:28,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:28,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:28,699 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:28,699 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:28,699 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:28,699 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:28,699 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,699 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,700 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:28,700 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:28,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:28,701 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,701 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:28,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:28,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:28,702 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:28,702 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:28,702 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:28,702 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:28,702 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:28,702 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:28,702 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:28,702 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:28,702 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:28,703 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:28,703 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:28,703 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,703 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:28,703 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:28,704 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,704 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:28,705 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:28,705 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:28,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:28,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56bd1af0[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:28,707 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:28,708 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,710 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,710 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,710 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:28,710 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:28,710 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:28,710 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:28,719 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,719 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:28,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:28,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:28,719 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:28,719 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:28,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:28,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:28,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:28,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:28,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:28,724 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:28,724 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:28,724 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:28,724 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:28,724 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:28,725 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,725 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,725 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:28,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:28,726 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,727 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,730 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:28,730 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:28,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:28,730 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,730 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,730 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,730 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,730 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:28,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:28,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:28,730 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:28,730 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:28,730 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,730 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,731 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,731 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:28,731 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:28,752 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=131, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/1c3eb3bc88ce460c96db5a08cb8e6ab7 2023-04-21 01:55:28,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:28,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38716 deadline: 1682042188752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,831 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:28,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:28,833 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:28,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:28,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:28,835 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:28,835 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:28,835 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:28,835 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:28,835 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,835 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,837 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:28,837 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:28,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:28,838 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,838 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:28,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:28,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:28,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:28,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:28,839 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:28,839 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:28,839 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:28,839 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:28,839 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:28,839 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:28,839 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:28,839 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:28,839 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:28,839 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:28,839 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:28,839 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,840 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:28,840 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:28,842 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,842 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:28,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:28,842 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,843 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:28,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:28,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e33e4c0[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:28,844 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:28,844 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,845 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,845 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:28,845 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:28,845 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:28,845 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,845 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:28,846 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:28,846 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:28,846 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,846 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:28,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:28,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:28,849 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:28,849 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:28,849 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:28,849 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:28,849 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:28,854 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,854 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:28,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,854 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:28,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:28,854 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:28,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:28,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,855 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:28,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,857 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,857 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,857 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,857 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,857 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,858 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,858 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:28,858 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:28,858 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:28,858 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,858 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:28,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:28,858 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:28,858 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:28,858 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:28,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:28,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:28,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:28,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:28,958 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:28,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:28,960 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:28,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:28,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:28,962 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:28,962 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:28,962 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:28,962 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:28,962 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,962 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,963 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:28,963 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:28,963 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,963 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:28,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:28,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:28,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:28,965 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:28,965 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:28,965 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:28,965 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:28,965 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:28,965 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:28,965 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:28,965 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:28,965 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:28,965 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:28,965 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:28,965 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,965 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:28,965 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:28,967 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,967 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,967 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:28,968 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:28,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:28,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7691f2a3[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:28,969 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:28,969 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,972 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,972 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:28,972 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:28,972 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:28,972 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,972 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:28,973 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:28,973 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:28,973 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,973 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:28,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:28,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:28,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:28,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:28,976 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:28,976 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:28,976 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:28,976 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:28,976 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:28,977 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,977 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:28,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:28,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,977 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:28,977 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:28,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,979 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,980 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,980 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:28,980 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:28,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:28,980 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:28,981 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:28,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:28,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:28,981 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:28,981 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:28,981 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:28,981 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:28,981 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,981 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:28,981 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:28,981 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:28,981 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:28,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:29,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:29,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38700 deadline: 1682042189054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,059 DEBUG [Thread-2529] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:55:29,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:29,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682042189057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,064 DEBUG [Thread-2533] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:55:29,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:29,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682042189065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,072 DEBUG [Thread-2527] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:55:29,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:29,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38730 deadline: 1682042189076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,082 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:29,083 DEBUG [Thread-2525] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:55:29,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:29,085 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:29,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:29,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:29,085 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:29,086 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:29,086 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:29,086 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:29,086 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,086 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:29,088 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:29,088 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:29,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:29,088 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,088 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:29,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:29,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:29,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:29,089 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:29,089 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:29,089 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:29,089 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:29,089 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:29,089 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:29,089 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:29,089 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:29,089 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:29,089 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:29,090 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:29,090 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:29,090 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:29,090 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:29,090 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,090 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:29,090 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:29,091 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,091 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:29,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:29,092 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,092 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:29,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:29,093 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:29,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f5c5911[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:29,093 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,094 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,094 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:29,094 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:29,094 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:29,094 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,094 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:29,095 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,095 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:29,095 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:29,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:29,095 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:29,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:29,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:29,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:29,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:29,098 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:29,098 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:29,098 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:29,098 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:29,098 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:29,103 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,103 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,103 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:29,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:29,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:29,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:29,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:29,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:29,103 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:29,104 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,106 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,106 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:29,106 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:29,106 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:29,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:29,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:29,106 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:29,107 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:29,106 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:29,106 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:29,107 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,107 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,107 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,107 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:29,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:29,107 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:29,107 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:29,165 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/27c87c2b63804b78808c4150d02f0983 2023-04-21 01:55:29,177 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/1e1ac9d19e30442c914e1d9754343be3 2023-04-21 01:55:29,181 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/1c3eb3bc88ce460c96db5a08cb8e6ab7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/1c3eb3bc88ce460c96db5a08cb8e6ab7 2023-04-21 01:55:29,185 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/1c3eb3bc88ce460c96db5a08cb8e6ab7, entries=200, sequenceid=131, filesize=32.0 K 2023-04-21 01:55:29,186 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/27c87c2b63804b78808c4150d02f0983 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/27c87c2b63804b78808c4150d02f0983 2023-04-21 01:55:29,196 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/27c87c2b63804b78808c4150d02f0983, entries=150, sequenceid=131, filesize=11.7 K 2023-04-21 01:55:29,197 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/1e1ac9d19e30442c914e1d9754343be3 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/1e1ac9d19e30442c914e1d9754343be3 2023-04-21 01:55:29,200 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/1e1ac9d19e30442c914e1d9754343be3, entries=150, sequenceid=131, filesize=11.7 K 2023-04-21 01:55:29,201 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 810791807ce3c9f7e9f14187121bf792 in 1267ms, sequenceid=131, compaction requested=true 2023-04-21 01:55:29,201 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:29,201 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:29,201 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:29,201 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:29,201 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:29,201 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:29,202 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:29,202 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 810791807ce3c9f7e9f14187121bf792/A is initiating minor compaction (all files) 2023-04-21 01:55:29,202 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 810791807ce3c9f7e9f14187121bf792/A in TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:29,202 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/551df64958b94bb5b4d2a214ab89510e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/7d3cd9326f194cc79647a55b07418acf, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/1c3eb3bc88ce460c96db5a08cb8e6ab7] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp, totalSize=82.5 K 2023-04-21 01:55:29,203 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35964 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:29,203 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 810791807ce3c9f7e9f14187121bf792/B is initiating minor compaction (all files) 2023-04-21 01:55:29,203 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 810791807ce3c9f7e9f14187121bf792/B in TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:29,203 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 551df64958b94bb5b4d2a214ab89510e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1682042124915 2023-04-21 01:55:29,203 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/7450abb6ed034cc0924c6a91ce9ae8a1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/5f02dbae1d4e45d0859ba6e5dbf69b4d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/27c87c2b63804b78808c4150d02f0983] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp, totalSize=35.1 K 2023-04-21 01:55:29,204 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 7450abb6ed034cc0924c6a91ce9ae8a1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1682042124915 2023-04-21 01:55:29,204 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 7d3cd9326f194cc79647a55b07418acf, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682042125687 2023-04-21 01:55:29,204 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 5f02dbae1d4e45d0859ba6e5dbf69b4d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682042125687 2023-04-21 01:55:29,204 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 1c3eb3bc88ce460c96db5a08cb8e6ab7, keycount=200, bloomtype=ROW, size=32.0 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1682042127901 2023-04-21 01:55:29,205 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 27c87c2b63804b78808c4150d02f0983, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1682042127901 2023-04-21 01:55:29,207 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:29,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:29,210 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:29,215 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 810791807ce3c9f7e9f14187121bf792#B#compaction#360 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:29,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:29,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:29,225 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:29,226 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:29,226 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:29,226 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:29,227 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,227 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:29,228 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:29,228 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:29,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:29,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:29,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:29,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:29,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:29,230 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,230 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:29,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:29,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:29,252 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:29,252 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:29,252 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:29,259 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/ef0eeabe9a7948e0a16b18df508cce51 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/ef0eeabe9a7948e0a16b18df508cce51 2023-04-21 01:55:29,262 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:29,262 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:29,262 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:29,262 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:29,262 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 810791807ce3c9f7e9f14187121bf792 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-21 01:55:29,263 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=A 2023-04-21 01:55:29,263 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:29,263 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=B 2023-04-21 01:55:29,263 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:29,263 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=C 2023-04-21 01:55:29,263 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:29,267 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 810791807ce3c9f7e9f14187121bf792#A#compaction#361 average throughput is 0.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:29,268 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 810791807ce3c9f7e9f14187121bf792/B of 810791807ce3c9f7e9f14187121bf792 into ef0eeabe9a7948e0a16b18df508cce51(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:29,268 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:29,268 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., storeName=810791807ce3c9f7e9f14187121bf792/B, priority=13, startTime=1682042129201; duration=0sec 2023-04-21 01:55:29,268 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:29,268 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:29,270 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35964 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:29,270 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 810791807ce3c9f7e9f14187121bf792/C is initiating minor compaction (all files) 2023-04-21 01:55:29,270 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 810791807ce3c9f7e9f14187121bf792/C in TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:29,270 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/45c5728448024b9b96ba0ebedd40abdf, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/44f43da7d4ed48e1b6c3f2225ca7b036, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/1e1ac9d19e30442c914e1d9754343be3] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp, totalSize=35.1 K 2023-04-21 01:55:29,270 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 45c5728448024b9b96ba0ebedd40abdf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1682042124915 2023-04-21 01:55:29,271 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 44f43da7d4ed48e1b6c3f2225ca7b036, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682042125687 2023-04-21 01:55:29,272 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 1e1ac9d19e30442c914e1d9754343be3, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1682042127901 2023-04-21 01:55:29,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:29,272 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:29,289 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 810791807ce3c9f7e9f14187121bf792#C#compaction#363 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:29,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:29,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38716 deadline: 1682042189311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:29,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38716 deadline: 1682042189421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:29,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38716 deadline: 1682042189629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,709 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/709547f0c103403abdbbdda3348dbc64 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/709547f0c103403abdbbdda3348dbc64 2023-04-21 01:55:29,714 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042116f5c6b613364f019cc9b0808471048e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042116f5c6b613364f019cc9b0808471048e 2023-04-21 01:55:29,718 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/4213cd5e78e14a5695a52d2231cfa5eb as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/4213cd5e78e14a5695a52d2231cfa5eb 2023-04-21 01:55:29,719 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 810791807ce3c9f7e9f14187121bf792/C of 810791807ce3c9f7e9f14187121bf792 into 709547f0c103403abdbbdda3348dbc64(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:29,720 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:29,720 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., storeName=810791807ce3c9f7e9f14187121bf792/C, priority=13, startTime=1682042129201; duration=0sec 2023-04-21 01:55:29,720 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:29,722 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 810791807ce3c9f7e9f14187121bf792/A of 810791807ce3c9f7e9f14187121bf792 into 4213cd5e78e14a5695a52d2231cfa5eb(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:29,722 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:29,722 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., storeName=810791807ce3c9f7e9f14187121bf792/A, priority=13, startTime=1682042129201; duration=0sec 2023-04-21 01:55:29,722 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:29,725 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=168, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/2729aeaafd3e435384e3d550d7eed8d3 2023-04-21 01:55:29,742 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/71caafa4f887495f87a52dad3060c1f0 2023-04-21 01:55:29,777 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/23444fbbcd444b2eb310e4d80274bc6d 2023-04-21 01:55:29,782 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/2729aeaafd3e435384e3d550d7eed8d3 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/2729aeaafd3e435384e3d550d7eed8d3 2023-04-21 01:55:29,795 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/2729aeaafd3e435384e3d550d7eed8d3, entries=150, sequenceid=168, filesize=25.3 K 2023-04-21 01:55:29,796 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/71caafa4f887495f87a52dad3060c1f0 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/71caafa4f887495f87a52dad3060c1f0 2023-04-21 01:55:29,806 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/71caafa4f887495f87a52dad3060c1f0, entries=150, sequenceid=168, filesize=11.8 K 2023-04-21 01:55:29,807 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/23444fbbcd444b2eb310e4d80274bc6d as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/23444fbbcd444b2eb310e4d80274bc6d 2023-04-21 01:55:29,810 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/23444fbbcd444b2eb310e4d80274bc6d, entries=150, sequenceid=168, filesize=11.8 K 2023-04-21 01:55:29,811 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 810791807ce3c9f7e9f14187121bf792 in 549ms, sequenceid=168, compaction requested=false 2023-04-21 01:55:29,811 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:29,811 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:29,811 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:29,811 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:29,811 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,811 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:29,811 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:29,815 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,815 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:29,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:29,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:29,816 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,816 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:29,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:29,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:29,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:29,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:29,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5108b490[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:29,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:29,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,821 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,821 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:29,821 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:29,821 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:29,821 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,821 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:29,823 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:29,823 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:29,823 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:29,824 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:29,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:29,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:29,827 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:29,827 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:29,827 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:29,827 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:29,827 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:29,828 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:29,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:29,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:29,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:29,829 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,829 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:29,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:29,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:29,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:29,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:29,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,830 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,831 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:29,833 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,833 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:29,833 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:29,833 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,833 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,833 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:29,833 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:29,833 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:29,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:29,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:29,834 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:29,833 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,834 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:29,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:29,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:29,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:29,934 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:29,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:29,936 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:29,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:29,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:29,939 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:29,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:29,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:29,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:29,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:29,940 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:29,940 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:29,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:29,941 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:29,941 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:29,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:29,941 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 810791807ce3c9f7e9f14187121bf792 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-21 01:55:29,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,941 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=A 2023-04-21 01:55:29,941 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:29,941 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=B 2023-04-21 01:55:29,941 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:29,942 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=C 2023-04-21 01:55:29,942 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:29,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:29,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:29,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:29,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:29,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:29,943 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:29,943 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:29,943 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:29,944 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:29,944 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:29,944 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:29,944 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:29,944 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:29,944 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:29,944 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:29,944 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:29,944 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,946 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:29,946 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:29,947 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,948 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,948 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:29,952 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:29,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:29,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:29,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:29,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:29,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:29,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:29,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@307c5f95[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:29,956 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:29,956 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,957 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,957 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:29,957 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:29,957 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:29,957 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,958 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:29,960 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:29,960 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:29,960 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:29,960 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:29,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:29,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:29,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:29,964 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:29,964 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:29,964 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:29,964 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:29,964 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:29,965 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,965 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:29,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,965 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:29,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:29,966 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,967 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:29,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:29,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,970 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,970 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:29,970 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:29,970 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:29,970 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:29,970 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:29,970 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:29,970 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:29,970 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:29,970 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,970 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:29,970 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:29,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:29,970 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:29,970 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:29,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:29,982 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042148b20b0b16ee4e65937eee1ebc358dea to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042148b20b0b16ee4e65937eee1ebc358dea 2023-04-21 01:55:29,998 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=183, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/caf6e2967e984825b79fbf58003128d5 2023-04-21 01:55:30,070 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:30,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:30,073 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:30,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:30,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:30,074 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:30,074 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:30,074 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:30,074 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:30,075 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,075 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:30,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042190070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,076 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:30,076 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:30,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:30,077 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,077 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:30,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:30,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:30,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:30,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:30,078 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:30,078 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:30,079 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:30,079 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:30,079 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:30,079 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:30,079 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:30,079 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:30,079 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:30,080 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:30,080 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:30,080 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,080 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:30,080 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:30,082 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,082 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:30,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:30,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:30,084 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,084 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:30,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:30,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:30,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4981a7d3[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:30,085 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:30,085 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,087 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,087 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:30,087 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:30,087 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,087 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:30,087 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:30,088 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:30,088 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:30,088 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,088 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:30,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:30,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:30,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:30,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:30,090 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:30,091 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:30,091 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:30,091 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:30,091 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:30,092 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,092 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:30,092 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:30,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:30,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:30,093 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:30,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:30,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,094 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,099 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,099 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,099 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,099 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:30,099 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,099 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:30,099 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:30,099 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:30,099 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:30,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:30,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:30,100 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:30,100 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:30,099 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:30,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:30,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:30,100 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,100 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:30,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042190177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,200 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:30,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:30,202 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:30,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:30,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:30,204 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:30,204 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:30,204 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:30,204 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:30,205 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,205 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,206 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,206 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,206 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:30,208 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:30,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:30,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:30,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:30,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,209 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:30,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:30,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:30,210 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:30,210 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:30,210 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:30,212 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:30,212 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:30,212 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:30,212 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:30,212 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:30,212 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:30,212 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:30,212 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:30,212 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,212 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:30,212 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:30,216 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,216 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,216 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:30,217 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:30,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:30,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:30,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:30,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:30,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:30,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4418eb23[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:30,219 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:30,219 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,220 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,220 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:30,220 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,221 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:30,221 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:30,221 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:30,223 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:30,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:30,224 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:30,224 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:30,224 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:30,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:30,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:30,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:30,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:30,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:30,226 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:30,226 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:30,226 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:30,226 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:30,226 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:30,227 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,227 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:30,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,228 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:30,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:30,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:30,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:30,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:30,228 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:30,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,231 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:30,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,233 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,233 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,233 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,233 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:30,233 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:30,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:30,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:30,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:30,233 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:30,233 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,233 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:30,234 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:30,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:30,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:30,233 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,234 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,234 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,234 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:30,234 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:30,334 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:30,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:30,336 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:30,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:30,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:30,337 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:30,337 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:30,337 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:30,337 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:30,338 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,338 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,339 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,339 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:30,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:30,339 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,339 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:30,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:30,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:30,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:30,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:30,341 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:30,341 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:30,341 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:30,341 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:30,341 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:30,341 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:30,341 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:30,341 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:30,341 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:30,341 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:30,341 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:30,341 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,341 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:30,341 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:30,343 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,343 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:30,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:30,344 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:30,344 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:30,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:30,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:30,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:30,345 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:30,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50207b04[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:30,345 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,346 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,346 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:30,346 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:30,346 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:30,346 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,346 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:30,347 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,347 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:30,347 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:30,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:30,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:30,348 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:30,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:30,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:30,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:30,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:30,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:30,350 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:30,350 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:30,350 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:30,350 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:30,350 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:30,352 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,352 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:30,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,352 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:30,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:30,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:30,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:30,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:30,360 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:30,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:30,363 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,366 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,366 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:30,366 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,366 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:30,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:30,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:30,366 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,366 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,366 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,366 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,366 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,366 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:30,366 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:30,367 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:30,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:30,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:30,367 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:30,367 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:30,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:30,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:30,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042190381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,416 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/4001f5ae3f0d4dba99f7a526a52efc6a 2023-04-21 01:55:30,467 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:30,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:30,469 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:30,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:30,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:30,470 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:30,470 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:30,470 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:30,470 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:30,471 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,471 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,472 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,472 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:30,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:30,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:30,472 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,472 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:30,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:30,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:30,474 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:30,474 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:30,474 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:30,474 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:30,474 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:30,474 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:30,474 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:30,474 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:30,474 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:30,474 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:30,475 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:30,475 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:30,475 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:30,475 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,475 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:30,475 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:30,476 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,476 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:30,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:30,476 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,476 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:30,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:30,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:30,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:30,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:30,478 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:30,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f21662[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:30,478 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,479 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,479 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:30,479 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:30,479 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,479 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:30,479 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:30,481 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:30,481 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:30,481 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,481 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:30,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:30,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:30,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:30,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:30,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:30,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:30,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:30,484 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:30,484 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:30,484 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:30,484 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:30,484 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:30,485 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,485 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,485 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:30,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:30,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:30,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,486 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:30,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:30,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:30,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:30,487 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:30,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,491 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:30,492 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:30,492 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:30,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:30,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:30,492 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,492 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:30,492 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:30,492 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:30,492 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:30,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:30,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:30,492 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,492 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:30,492 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,492 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,492 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,493 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,592 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:30,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:30,595 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:30,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:30,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:30,597 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:30,597 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:30,597 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:30,597 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:30,597 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,597 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,600 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:30,600 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:30,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:30,600 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,600 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:30,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:30,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:30,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:30,601 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:30,601 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:30,601 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:30,602 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:30,602 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:30,602 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:30,602 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:30,602 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:30,602 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:30,602 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:30,602 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:30,602 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,602 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:30,602 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:30,605 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,605 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:30,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:30,605 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,605 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:30,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:30,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:30,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:30,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:30,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@460986d0[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:30,606 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:30,607 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,607 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,608 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:30,608 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:30,608 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,608 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:30,608 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:30,609 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,609 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:30,609 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:30,610 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:30,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:30,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:30,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:30,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:30,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:30,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:30,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:30,612 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:30,612 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:30,612 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:30,612 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:30,612 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:30,615 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:30,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:30,615 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,615 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:30,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:30,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:30,616 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,617 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:30,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:30,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,619 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:30,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,619 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:30,619 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:30,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:30,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:30,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:30,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:30,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,619 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:30,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,619 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:30,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:30,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:30,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,620 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:30,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042190688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,719 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:30,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:30,721 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:30,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:30,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:30,722 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:30,722 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:30,723 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:30,723 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:30,723 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,723 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,724 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:30,724 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:30,724 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,724 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:30,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:30,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,725 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:30,725 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:30,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:30,726 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:30,726 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:30,726 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:30,726 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:30,726 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:30,726 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:30,726 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:30,726 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:30,726 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:30,726 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:30,726 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:30,726 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,726 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:30,726 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:30,728 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,728 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:30,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:30,728 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,728 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:30,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:30,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:30,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:30,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:30,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45cb2cc1[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:30,730 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:30,730 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,732 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,732 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:30,732 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:30,732 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:30,732 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,732 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:30,733 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,733 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:30,733 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:30,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:30,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:30,733 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:30,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:30,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:30,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:30,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:30,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:30,736 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:30,736 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:30,736 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:30,736 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:30,736 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:30,737 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,737 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,737 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:30,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:30,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:30,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:30,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:30,738 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:30,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:30,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,739 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:30,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,741 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:30,741 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:30,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:30,741 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:30,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:30,741 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:30,741 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:30,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:30,741 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,741 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,741 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,741 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,742 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:30,742 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:30,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:30,742 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:30,742 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,742 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,838 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/7968bac842a14e69850dd866d4df124a 2023-04-21 01:55:30,842 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:30,842 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/caf6e2967e984825b79fbf58003128d5 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/caf6e2967e984825b79fbf58003128d5 2023-04-21 01:55:30,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:30,845 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:30,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:30,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:30,846 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:30,846 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:30,846 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:30,846 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:30,846 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/caf6e2967e984825b79fbf58003128d5, entries=200, sequenceid=183, filesize=32.1 K 2023-04-21 01:55:30,846 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,847 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,847 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/4001f5ae3f0d4dba99f7a526a52efc6a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/4001f5ae3f0d4dba99f7a526a52efc6a 2023-04-21 01:55:30,849 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:30,849 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:30,850 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,850 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:30,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:30,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:30,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:30,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:30,851 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:30,851 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:30,851 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:30,851 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:30,851 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:30,852 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:30,852 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:30,852 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:30,852 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:30,852 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:30,852 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:30,852 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,852 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:30,852 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:30,853 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/4001f5ae3f0d4dba99f7a526a52efc6a, entries=150, sequenceid=183, filesize=11.8 K 2023-04-21 01:55:30,853 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,853 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:30,854 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,854 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:30,854 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/7968bac842a14e69850dd866d4df124a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/7968bac842a14e69850dd866d4df124a 2023-04-21 01:55:30,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:30,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:30,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:30,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16e08727[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:30,855 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:30,855 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,856 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,857 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:30,857 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:30,857 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:30,857 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,857 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:30,858 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/7968bac842a14e69850dd866d4df124a, entries=150, sequenceid=183, filesize=11.8 K 2023-04-21 01:55:30,859 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 810791807ce3c9f7e9f14187121bf792 in 917ms, sequenceid=183, compaction requested=true 2023-04-21 01:55:30,859 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:30,859 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:30,859 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:30,859 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:30,859 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:30,859 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:30,860 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,860 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:30,860 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:30,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:30,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:30,860 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:30,860 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84936 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:30,860 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 810791807ce3c9f7e9f14187121bf792/A is initiating minor compaction (all files) 2023-04-21 01:55:30,860 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 810791807ce3c9f7e9f14187121bf792/A in TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:30,860 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/4213cd5e78e14a5695a52d2231cfa5eb, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/2729aeaafd3e435384e3d550d7eed8d3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/caf6e2967e984825b79fbf58003128d5] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp, totalSize=82.9 K 2023-04-21 01:55:30,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:30,861 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36366 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:30,861 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 810791807ce3c9f7e9f14187121bf792/B is initiating minor compaction (all files) 2023-04-21 01:55:30,861 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 4213cd5e78e14a5695a52d2231cfa5eb, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1682042127901 2023-04-21 01:55:30,861 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 810791807ce3c9f7e9f14187121bf792/B in TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:30,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:30,861 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/ef0eeabe9a7948e0a16b18df508cce51, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/71caafa4f887495f87a52dad3060c1f0, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/4001f5ae3f0d4dba99f7a526a52efc6a] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp, totalSize=35.5 K 2023-04-21 01:55:30,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,861 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 2729aeaafd3e435384e3d550d7eed8d3, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1682042128105 2023-04-21 01:55:30,861 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting ef0eeabe9a7948e0a16b18df508cce51, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1682042127901 2023-04-21 01:55:30,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,862 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting caf6e2967e984825b79fbf58003128d5, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1682042129297 2023-04-21 01:55:30,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:30,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,863 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 71caafa4f887495f87a52dad3060c1f0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1682042128105 2023-04-21 01:55:30,863 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 4001f5ae3f0d4dba99f7a526a52efc6a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1682042129297 2023-04-21 01:55:30,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:30,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:30,863 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:30,863 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:30,863 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:30,863 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:30,863 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:30,866 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:30,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:30,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:30,867 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:30,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,870 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:30,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,872 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,872 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:30,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:30,872 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,873 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:30,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:30,873 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:30,873 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:30,873 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,873 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,873 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,873 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,873 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:30,873 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,873 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:30,873 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:30,874 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:30,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:30,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:30,889 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 810791807ce3c9f7e9f14187121bf792#B#compaction#369 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:30,909 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 810791807ce3c9f7e9f14187121bf792#A#compaction#370 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:30,973 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:30,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:30,975 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:30,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:30,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:30,976 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:30,977 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:30,977 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:30,977 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:30,977 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,977 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,978 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,978 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:30,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:30,979 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:30,979 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:30,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:30,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:30,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:30,980 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:30,980 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:30,980 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:30,980 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:30,980 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:30,980 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:30,980 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:30,981 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 810791807ce3c9f7e9f14187121bf792 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-21 01:55:30,981 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=A 2023-04-21 01:55:30,981 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:30,981 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=B 2023-04-21 01:55:30,981 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:30,981 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=C 2023-04-21 01:55:30,981 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:31,002 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304215a92ac26f0c241418354f0c3ca58147c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304215a92ac26f0c241418354f0c3ca58147c 2023-04-21 01:55:31,008 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=203, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/c7ef206510e041048387afb007bcd09f 2023-04-21 01:55:31,018 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/8f95991f4a044166baf749083b25b26d 2023-04-21 01:55:31,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:31,197 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:31,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:31,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042191250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,316 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/4bd2840d302549fa8953c2c154ef2182 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/4bd2840d302549fa8953c2c154ef2182 2023-04-21 01:55:31,321 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 810791807ce3c9f7e9f14187121bf792/B of 810791807ce3c9f7e9f14187121bf792 into 4bd2840d302549fa8953c2c154ef2182(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:31,321 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:31,321 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., storeName=810791807ce3c9f7e9f14187121bf792/B, priority=13, startTime=1682042130859; duration=0sec 2023-04-21 01:55:31,321 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:31,321 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:31,322 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36366 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:31,322 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 810791807ce3c9f7e9f14187121bf792/C is initiating minor compaction (all files) 2023-04-21 01:55:31,322 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 810791807ce3c9f7e9f14187121bf792/C in TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:31,322 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/709547f0c103403abdbbdda3348dbc64, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/23444fbbcd444b2eb310e4d80274bc6d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/7968bac842a14e69850dd866d4df124a] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp, totalSize=35.5 K 2023-04-21 01:55:31,322 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 709547f0c103403abdbbdda3348dbc64, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1682042127901 2023-04-21 01:55:31,323 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 23444fbbcd444b2eb310e4d80274bc6d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1682042128105 2023-04-21 01:55:31,323 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 7968bac842a14e69850dd866d4df124a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1682042129297 2023-04-21 01:55:31,328 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 810791807ce3c9f7e9f14187121bf792#C#compaction#374 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:31,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:31,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042191352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,430 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/3afd20756c8d49d9b1a1b159db16a627 2023-04-21 01:55:31,434 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/c7ef206510e041048387afb007bcd09f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/c7ef206510e041048387afb007bcd09f 2023-04-21 01:55:31,440 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/c7ef206510e041048387afb007bcd09f, entries=150, sequenceid=203, filesize=25.3 K 2023-04-21 01:55:31,441 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/8f95991f4a044166baf749083b25b26d as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/8f95991f4a044166baf749083b25b26d 2023-04-21 01:55:31,445 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/8f95991f4a044166baf749083b25b26d, entries=150, sequenceid=203, filesize=11.8 K 2023-04-21 01:55:31,446 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/3afd20756c8d49d9b1a1b159db16a627 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/3afd20756c8d49d9b1a1b159db16a627 2023-04-21 01:55:31,449 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/3afd20756c8d49d9b1a1b159db16a627, entries=150, sequenceid=203, filesize=11.8 K 2023-04-21 01:55:31,450 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 810791807ce3c9f7e9f14187121bf792 in 469ms, sequenceid=203, compaction requested=false 2023-04-21 01:55:31,450 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:31,450 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:31,450 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:31,450 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:31,450 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,450 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:31,450 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:31,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,455 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:31,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:31,456 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,456 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:31,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:31,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:31,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:31,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:31,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b3d29f[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:31,458 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:31,458 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,459 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,459 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:31,459 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,459 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:31,459 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:31,459 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:31,460 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:31,460 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:31,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,460 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:31,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:31,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:31,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:31,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:31,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:31,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:31,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:31,463 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:31,463 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:31,463 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:31,463 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:31,463 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:31,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:31,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:31,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:31,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:31,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:31,465 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:31,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:31,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,466 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:31,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,470 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,470 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:31,470 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:31,470 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:31,470 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:31,470 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:31,470 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:31,470 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:31,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:31,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:31,470 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:31,470 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,470 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:31,470 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,470 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,471 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:31,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:31,571 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:31,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:31,573 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:31,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:31,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:31,588 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:31,588 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:31,588 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:31,588 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:31,589 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,589 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:31,590 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,590 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:31,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:31,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:31,590 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,590 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:31,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:31,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:31,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:31,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:31,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:31,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:31,592 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:31,592 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:31,592 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:31,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:31,595 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 810791807ce3c9f7e9f14187121bf792 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-21 01:55:31,595 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=A 2023-04-21 01:55:31,595 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:31,595 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=B 2023-04-21 01:55:31,595 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:31,595 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=C 2023-04-21 01:55:31,595 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:31,604 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:31,604 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:31,604 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:31,604 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:31,605 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:31,605 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:31,605 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:31,605 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:31,605 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,605 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:31,605 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:31,606 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:31,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:31,607 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,612 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,612 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:31,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:31,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:31,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:31,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:31,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e06fbdf[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:31,615 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:31,616 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,617 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,617 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:31,617 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:31,617 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:31,617 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,617 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:31,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,619 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:31,619 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:31,619 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:31,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:31,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:31,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:31,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:31,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:31,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:31,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:31,622 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:31,622 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:31,622 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:31,622 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:31,622 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:31,624 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,624 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:31,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:31,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,624 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:31,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:31,625 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:31,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:31,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:31,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,627 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,630 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:31,630 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:31,630 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:31,630 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:31,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:31,630 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:31,630 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:31,630 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:31,630 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,630 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,630 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,630 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,630 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:31,630 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:31,631 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:31,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:31,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:31,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:31,729 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/0c0a443bf26e4091bbdc8c4fd94feb6c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/0c0a443bf26e4091bbdc8c4fd94feb6c 2023-04-21 01:55:31,731 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:31,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:31,733 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:31,733 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 810791807ce3c9f7e9f14187121bf792/A of 810791807ce3c9f7e9f14187121bf792 into 0c0a443bf26e4091bbdc8c4fd94feb6c(size=25.7 K), total size for store is 51.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:31,733 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:31,733 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., storeName=810791807ce3c9f7e9f14187121bf792/A, priority=13, startTime=1682042130859; duration=0sec 2023-04-21 01:55:31,733 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:31,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:31,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:31,734 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:31,734 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:31,735 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:31,735 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:31,735 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/5fa5a8a1b9004c518d23ba5b7dfc6d55 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/5fa5a8a1b9004c518d23ba5b7dfc6d55 2023-04-21 01:55:31,736 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,736 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:31,738 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,738 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:31,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:31,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:31,738 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,738 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:31,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:31,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:31,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:31,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:31,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:31,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:31,739 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:31,739 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:31,739 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:31,739 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:31,739 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:31,739 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:31,739 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:31,739 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:31,739 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 810791807ce3c9f7e9f14187121bf792/C of 810791807ce3c9f7e9f14187121bf792 into 5fa5a8a1b9004c518d23ba5b7dfc6d55(size=12.2 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:31,739 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:31,739 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:31,739 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:31,740 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:31,740 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., storeName=810791807ce3c9f7e9f14187121bf792/C, priority=13, startTime=1682042130859; duration=0sec 2023-04-21 01:55:31,740 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,740 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:31,740 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:31,740 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:31,742 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,742 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:31,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:31,742 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,742 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:31,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:31,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:31,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12b773fa[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:31,744 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:31,744 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,745 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,745 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:31,745 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:31,745 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:31,745 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,745 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:31,746 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,746 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:31,746 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:31,746 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:31,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:31,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:31,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:31,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:31,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:31,749 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:31,749 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:31,749 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:31,749 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:31,749 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:31,751 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,751 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:31,751 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:31,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:31,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:31,751 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:31,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,752 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,756 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:31,756 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:31,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:31,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:31,756 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:31,756 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:31,756 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:31,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:31,756 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:31,756 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:31,756 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:31,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:31,756 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:31,756 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:31,756 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,756 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,756 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,756 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:31,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042191772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,856 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:31,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:31,859 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:31,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:31,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:31,863 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:31,864 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:31,864 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:31,864 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:31,864 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,864 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:31,865 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:31,865 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:31,866 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:31,866 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:31,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:31,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:31,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:31,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:31,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:31,868 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:31,868 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:31,868 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:31,871 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:31,871 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:31,871 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:31,871 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:31,871 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:31,871 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:31,871 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:31,871 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:31,871 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,871 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:31,871 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:31,873 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:31,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:31,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,874 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,875 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,875 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:31,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:31,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:31,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fc1734[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:31,875 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:31,875 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,880 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:31,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:31,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:31,881 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,881 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:31,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:31,882 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:31,882 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:31,882 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:31,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:31,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:31,882 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:31,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042191877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:31,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:31,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:31,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:31,885 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:31,885 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:31,885 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:31,885 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:31,885 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:31,887 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,887 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,887 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:31,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:31,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:31,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:31,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:31,887 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:31,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:31,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,888 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:31,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:31,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,890 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:31,890 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:31,891 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:31,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:31,891 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:31,891 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:31,891 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,891 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:31,891 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:31,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:31,891 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:31,891 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:31,891 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:31,891 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,891 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:31,891 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:31,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:31,991 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:31,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:31,993 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:31,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:31,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:31,994 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:31,994 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:31,994 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:31,994 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:31,995 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,995 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:31,997 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,997 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:31,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:31,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:31,997 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:31,997 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:31,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:31,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:31,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:31,998 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:31,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:31,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:31,998 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:31,998 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:31,998 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:31,998 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:31,998 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:31,999 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:31,999 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:31,999 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:31,999 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:31,999 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:31,999 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:31,999 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:31,999 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:31,999 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:32,001 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,001 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,001 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,001 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:32,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ac10a9e[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:32,002 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:32,003 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,004 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,004 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:32,004 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:32,004 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:32,004 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,004 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:32,005 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,005 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:32,006 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,006 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:32,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:32,008 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:32,008 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:32,008 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:32,008 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:32,008 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:32,011 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,011 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,011 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:32,011 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,012 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,014 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,014 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,015 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,015 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:32,015 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:32,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:32,015 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:32,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:32,015 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:32,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:32,015 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:32,015 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,015 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,015 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,015 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,015 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:32,015 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:32,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:32,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:32,035 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042149c05de38762408cb7c131d9afe5f5e5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042149c05de38762408cb7c131d9afe5f5e5 2023-04-21 01:55:32,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:32,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042192084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,115 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:32,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:32,118 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:32,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:32,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:32,119 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:32,119 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:32,119 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:32,119 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:32,119 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,119 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,123 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:32,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:32,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:32,123 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,124 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,124 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:32,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:32,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:32,125 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:32,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:32,125 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:32,125 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:32,125 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:32,125 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:32,125 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:32,125 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:32,125 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:32,126 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:32,126 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:32,126 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:32,126 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:32,126 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,126 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:32,126 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:32,127 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,127 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,127 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,128 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:32,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:32,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2320c21a[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:32,129 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:32,129 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,131 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,131 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:32,131 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:32,131 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:32,131 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,131 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:32,132 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:32,132 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,132 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,132 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:32,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:32,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:32,137 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:32,138 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:32,138 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:32,138 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:32,138 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:32,139 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,139 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,139 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:32,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:32,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:32,140 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,142 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,144 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,144 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:32,144 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,144 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:32,144 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,144 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:32,144 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:32,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:32,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:32,145 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:32,144 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:32,145 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:32,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:32,144 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,145 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,145 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,145 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:32,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:32,245 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:32,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:32,247 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:32,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:32,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:32,248 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:32,248 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:32,248 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:32,248 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:32,248 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,248 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,250 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:32,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:32,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:32,250 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,251 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,251 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:32,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:32,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:32,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:32,253 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:32,253 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:32,253 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:32,253 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:32,255 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,255 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,256 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,256 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:32,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aa15d27[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:32,258 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:32,258 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,259 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,259 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:32,259 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:32,259 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:32,259 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,260 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:32,261 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,261 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,261 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:32,261 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:32,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:32,263 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:32,264 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:32,264 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:32,264 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:32,264 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:32,274 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,274 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,275 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:32,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:32,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:32,275 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,276 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,281 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,281 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,281 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,281 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:32,281 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:32,281 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:32,281 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:32,281 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:32,282 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:32,282 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:32,281 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:32,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:32,282 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,282 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,282 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:32,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:32,382 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:32,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:32,384 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:32,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:32,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:32,385 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:32,386 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:32,386 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:32,386 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:32,386 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,386 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,387 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:32,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:32,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:32,387 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,388 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,388 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:32,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:32,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:32,389 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:32,389 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:32,389 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:32,389 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:32,389 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:32,389 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:32,389 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:32,389 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:32,389 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:32,389 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:32,389 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:32,389 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:32,389 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:32,389 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,389 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:32,390 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:32,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:32,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042192388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,391 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,391 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,391 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,391 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:32,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:32,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a84aef0[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:32,392 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:32,393 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,394 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:32,395 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,395 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:32,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:32,395 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:32,397 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:32,397 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:32,397 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,397 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:32,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:32,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:32,399 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:32,399 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:32,399 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:32,399 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:32,399 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:32,401 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,401 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,401 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:32,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,401 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:32,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:32,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,404 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,406 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:32,406 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,406 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:32,406 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:32,406 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:32,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:32,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:32,406 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:32,406 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:32,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:32,406 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:32,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:32,406 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,406 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:32,406 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,407 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,407 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,407 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,440 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=217, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/3a7016e12ad146bdb49ddb05729e8297 2023-04-21 01:55:32,506 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:32,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:32,509 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:32,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:32,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:32,510 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:32,510 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:32,510 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:32,510 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:32,511 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,511 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,513 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:32,513 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:32,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:32,513 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,513 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:32,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:32,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:32,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:32,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:32,515 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:32,515 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:32,515 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:32,516 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:32,516 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:32,516 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:32,516 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:32,516 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:32,516 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:32,516 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:32,516 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:32,516 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,516 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:32,516 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:32,518 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,518 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,518 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,518 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:32,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7883ccc4[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:32,520 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:32,520 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,521 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,521 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:32,521 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,521 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:32,521 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:32,521 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:32,522 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,522 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:32,522 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:32,522 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:32,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:32,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:32,524 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:32,524 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:32,525 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:32,525 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:32,525 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:32,528 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,528 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,528 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:32,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,528 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:32,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:32,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,529 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,531 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,531 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,531 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,531 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:32,531 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:32,531 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:32,531 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,531 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,531 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,531 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,532 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:32,532 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:32,532 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:32,532 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:32,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:32,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:32,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:32,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:32,632 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:32,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:32,634 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:32,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:32,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:32,636 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:32,636 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:32,636 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:32,636 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:32,636 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,637 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,638 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:32,638 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,638 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,638 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:32,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:32,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:32,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:32,639 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:32,639 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:32,639 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:32,639 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:32,639 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:32,639 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:32,639 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:32,639 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:32,640 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:32,640 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:32,640 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:32,640 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,640 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:32,640 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:32,641 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,641 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,641 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,642 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:32,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:32,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d16f1dc[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:32,643 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:32,643 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,646 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,647 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:32,647 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:32,647 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:32,647 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,647 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:32,648 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,648 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:32,648 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:32,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,648 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:32,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:32,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:32,650 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:32,650 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:32,650 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:32,650 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:32,650 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:32,652 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,652 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,652 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:32,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:32,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:32,652 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,656 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:32,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:32,666 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:32,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:32,666 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:32,666 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:32,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:32,666 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:32,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:32,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:32,666 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:32,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:32,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,667 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,667 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,667 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,767 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:32,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:32,768 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:32,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:32,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:32,769 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:32,770 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:32,770 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:32,770 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:32,770 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,770 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,771 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,771 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:32,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:32,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:32,771 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,771 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:32,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:32,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,772 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:32,773 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:32,773 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:32,773 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:32,773 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:32,773 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:32,775 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:32,775 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:32,775 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:32,775 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:32,775 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:32,775 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:32,775 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:32,775 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:32,775 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,775 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:32,775 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:32,776 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,776 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,777 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,777 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:32,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:32,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d5ed73c[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:32,778 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:32,778 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,779 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,779 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:32,779 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:32,779 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,779 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:32,779 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:32,781 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,781 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:32,781 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:32,781 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:32,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:32,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:32,783 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:32,784 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:32,784 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:32,784 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:32,784 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:32,785 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,785 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,785 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:32,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:32,786 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:32,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,787 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,788 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,788 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,788 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:32,788 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,789 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:32,789 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:32,789 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,789 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,789 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,789 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,788 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:32,788 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:32,789 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:32,789 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:32,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:32,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:32,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:32,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:32,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:32,857 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/1ecdd21062c04250a62dff078558a0c7 2023-04-21 01:55:32,889 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:32,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:32,891 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:32,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:32,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:32,892 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:32,892 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:32,892 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:32,892 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:32,892 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,892 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,894 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,894 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:32,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:32,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:32,894 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,894 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:32,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042192893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:32,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:32,895 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:32,895 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:32,895 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:32,895 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:32,895 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:32,896 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:32,896 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:32,896 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:32,896 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:32,896 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:32,896 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:32,896 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:32,896 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,896 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:32,896 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:32,898 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,898 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,899 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,899 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:32,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:32,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12e20149[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:32,901 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:32,901 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,902 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,902 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:32,902 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:32,902 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,902 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:32,902 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:32,904 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,904 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:32,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,904 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:32,904 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:32,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:32,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:32,906 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:32,906 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:32,906 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:32,906 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:32,906 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:32,907 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,907 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,907 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:32,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:32,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:32,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:32,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:32,908 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:32,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:32,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,909 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:32,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:32,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,912 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,912 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,912 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:32,912 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:32,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:32,912 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:32,912 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:32,912 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:32,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:32,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:32,912 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:32,912 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:32,912 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,912 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:32,913 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:32,913 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:32,913 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:32,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:32,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:33,013 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:33,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:33,015 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:33,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:33,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:33,016 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:33,016 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:33,016 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:33,016 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:33,017 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,017 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:33,018 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:33,018 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:33,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:33,018 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,018 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:33,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:33,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:33,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:33,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:33,019 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:33,019 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:33,020 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:33,020 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:33,020 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:33,020 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:33,020 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:33,020 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:33,020 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:33,020 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:33,020 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:33,020 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:33,020 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:33,020 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,020 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:33,020 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:33,022 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,022 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:33,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:33,022 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,022 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:33,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:33,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:33,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:33,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:33,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:33,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74e10a3d[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:33,023 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:33,023 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,024 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,024 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:33,024 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:33,024 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:33,025 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,025 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:33,032 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:33,032 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:33,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:33,032 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:33,032 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:33,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:33,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:33,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:33,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:33,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:33,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:33,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:33,034 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:33,034 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:33,034 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:33,034 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:33,034 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:33,036 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,036 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,036 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:33,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:33,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:33,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:33,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:33,036 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:33,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:33,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:33,037 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:33,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:33,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:33,041 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:33,041 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,042 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:33,041 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:33,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:33,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:33,042 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:33,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:33,042 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:33,042 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:33,042 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:33,042 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,042 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,042 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,042 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:33,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:33,042 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:33,042 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:33,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:33,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38700 deadline: 1682042193063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,067 DEBUG [Thread-2529] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:55:33,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:33,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38690 deadline: 1682042193082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,088 DEBUG [Thread-2533] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8170 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:55:33,092 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:33,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38730 deadline: 1682042193089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,095 DEBUG [Thread-2525] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8174 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:55:33,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:33,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38654 deadline: 1682042193105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,111 DEBUG [Thread-2527] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8193 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:55:33,143 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:33,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:33,145 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:33,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:33,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:33,145 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:33,146 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:33,146 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:33,146 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:33,146 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,146 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:33,148 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,148 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:33,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:33,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:33,148 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,149 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:33,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:33,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:33,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:33,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:33,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:33,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:33,150 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:33,150 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:33,150 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:33,150 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:33,150 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:33,150 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:33,150 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:33,150 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:33,150 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:33,150 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:33,150 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:33,150 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,150 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:33,150 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:33,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:33,153 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:33,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:33,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:33,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:33,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:33,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:33,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:33,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e939d6c[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:33,154 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:33,154 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,155 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,156 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:33,156 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,156 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:33,156 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:33,156 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:33,157 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,157 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:33,157 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:33,157 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:33,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:33,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:33,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:33,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:33,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:33,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:33,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:33,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:33,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:33,160 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:33,160 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:33,160 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:33,160 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:33,160 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:33,162 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,162 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:33,162 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:33,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:33,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:33,163 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:33,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:33,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:33,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,164 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:33,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:33,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,166 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,166 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:33,166 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:33,166 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:33,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:33,166 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:33,166 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:33,166 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,166 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:33,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:33,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:33,166 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:33,166 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,167 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:33,167 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,167 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:33,167 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:33,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:33,267 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:33,267 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/96fb6f99f75d46c99973163a70a48637 2023-04-21 01:55:33,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:33,270 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:33,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:33,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:33,271 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:33,271 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:33,271 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:33,271 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:33,271 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,271 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:33,272 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/3a7016e12ad146bdb49ddb05729e8297 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/3a7016e12ad146bdb49ddb05729e8297 2023-04-21 01:55:33,273 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,273 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:33,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:33,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:33,273 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,273 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:33,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:33,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:33,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:33,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:33,274 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:33,274 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:33,274 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:33,274 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:33,274 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:33,274 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:33,274 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:33,275 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/3a7016e12ad146bdb49ddb05729e8297, entries=150, sequenceid=217, filesize=25.3 K 2023-04-21 01:55:33,276 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:33,276 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:33,276 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/1ecdd21062c04250a62dff078558a0c7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/1ecdd21062c04250a62dff078558a0c7 2023-04-21 01:55:33,276 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:33,276 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:33,277 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:33,277 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:33,277 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,277 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:33,277 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:33,279 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,279 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:33,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:33,279 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,279 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:33,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:33,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:33,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:33,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:33,281 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/1ecdd21062c04250a62dff078558a0c7, entries=100, sequenceid=217, filesize=9.4 K 2023-04-21 01:55:33,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:33,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35df4eb5[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:33,281 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:33,281 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,281 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/96fb6f99f75d46c99973163a70a48637 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/96fb6f99f75d46c99973163a70a48637 2023-04-21 01:55:33,285 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/96fb6f99f75d46c99973163a70a48637, entries=100, sequenceid=217, filesize=9.4 K 2023-04-21 01:55:33,286 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 810791807ce3c9f7e9f14187121bf792 in 1691ms, sequenceid=217, compaction requested=true 2023-04-21 01:55:33,286 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:33,286 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:33,286 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:33,286 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:33,286 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:33,287 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:33,287 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78094 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:33,288 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 810791807ce3c9f7e9f14187121bf792/A is initiating minor compaction (all files) 2023-04-21 01:55:33,288 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 810791807ce3c9f7e9f14187121bf792/A in TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:33,288 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34124 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:33,288 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/0c0a443bf26e4091bbdc8c4fd94feb6c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/c7ef206510e041048387afb007bcd09f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/3a7016e12ad146bdb49ddb05729e8297] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp, totalSize=76.3 K 2023-04-21 01:55:33,288 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 810791807ce3c9f7e9f14187121bf792/B is initiating minor compaction (all files) 2023-04-21 01:55:33,288 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 810791807ce3c9f7e9f14187121bf792/B in TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:33,288 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,288 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/4bd2840d302549fa8953c2c154ef2182, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/8f95991f4a044166baf749083b25b26d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/1ecdd21062c04250a62dff078558a0c7] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp, totalSize=33.3 K 2023-04-21 01:55:33,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,288 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:33,288 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:33,288 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,288 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:33,288 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:33,288 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 0c0a443bf26e4091bbdc8c4fd94feb6c, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1682042129297 2023-04-21 01:55:33,289 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 4bd2840d302549fa8953c2c154ef2182, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1682042129297 2023-04-21 01:55:33,289 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting c7ef206510e041048387afb007bcd09f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1682042130038 2023-04-21 01:55:33,289 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 8f95991f4a044166baf749083b25b26d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1682042130038 2023-04-21 01:55:33,289 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 3a7016e12ad146bdb49ddb05729e8297, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682042131242 2023-04-21 01:55:33,290 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 1ecdd21062c04250a62dff078558a0c7, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682042131242 2023-04-21 01:55:33,290 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:33,290 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:33,290 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:33,290 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:33,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:33,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:33,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:33,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:33,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:33,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:33,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:33,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:33,295 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:33,295 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:33,295 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:33,295 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:33,295 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:33,297 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:33,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:33,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:33,298 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:33,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,299 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:33,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,304 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:33,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:33,304 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:33,305 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:33,305 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:33,305 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:33,308 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,308 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:33,308 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:33,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,308 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,308 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:33,308 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:33,308 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,308 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,308 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:33,308 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:33,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:33,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:33,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:33,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:33,318 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 810791807ce3c9f7e9f14187121bf792#B#compaction#378 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:33,357 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 810791807ce3c9f7e9f14187121bf792#A#compaction#379 average throughput is 0.53 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:33,374 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/8c24803137694d338a2a4e94491c7285 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/8c24803137694d338a2a4e94491c7285 2023-04-21 01:55:33,381 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 810791807ce3c9f7e9f14187121bf792/B of 810791807ce3c9f7e9f14187121bf792 into 8c24803137694d338a2a4e94491c7285(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:33,381 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:33,381 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., storeName=810791807ce3c9f7e9f14187121bf792/B, priority=13, startTime=1682042133286; duration=0sec 2023-04-21 01:55:33,381 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:33,381 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:33,383 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34124 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:33,383 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 810791807ce3c9f7e9f14187121bf792/C is initiating minor compaction (all files) 2023-04-21 01:55:33,383 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 810791807ce3c9f7e9f14187121bf792/C in TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:33,383 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/5fa5a8a1b9004c518d23ba5b7dfc6d55, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/3afd20756c8d49d9b1a1b159db16a627, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/96fb6f99f75d46c99973163a70a48637] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp, totalSize=33.3 K 2023-04-21 01:55:33,384 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 5fa5a8a1b9004c518d23ba5b7dfc6d55, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1682042129297 2023-04-21 01:55:33,386 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 3afd20756c8d49d9b1a1b159db16a627, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1682042130038 2023-04-21 01:55:33,386 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 96fb6f99f75d46c99973163a70a48637, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682042131242 2023-04-21 01:55:33,398 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 810791807ce3c9f7e9f14187121bf792#C#compaction#380 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:33,405 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:33,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:33,410 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:33,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:33,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:33,412 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:33,412 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:33,412 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:33,412 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:33,412 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,413 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:33,414 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,414 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:33,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:33,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:33,415 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:33,415 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:33,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:33,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:33,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:33,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:33,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:33,420 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/559258a7590e4fffad960d82927d6455 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/559258a7590e4fffad960d82927d6455 2023-04-21 01:55:33,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:33,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:33,425 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:33,425 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:33,425 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:33,429 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 810791807ce3c9f7e9f14187121bf792/C of 810791807ce3c9f7e9f14187121bf792 into 559258a7590e4fffad960d82927d6455(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:33,429 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:33,429 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., storeName=810791807ce3c9f7e9f14187121bf792/C, priority=13, startTime=1682042133286; duration=0sec 2023-04-21 01:55:33,430 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:33,433 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:33,433 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:33,433 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:33,433 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:33,433 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 810791807ce3c9f7e9f14187121bf792 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-21 01:55:33,433 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=A 2023-04-21 01:55:33,433 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:33,433 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=B 2023-04-21 01:55:33,433 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:33,434 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=C 2023-04-21 01:55:33,434 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:33,864 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421486ce04eae2f49a8a8525a691aeb9a4e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421486ce04eae2f49a8a8525a691aeb9a4e 2023-04-21 01:55:33,869 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=240, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/c3d83ca586c3463f9ee666d0b8d1fcb8 2023-04-21 01:55:33,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:33,900 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:33,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:33,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042193964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:34,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:34,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042194072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:34,209 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/6ee13b1f362647948c98ec0833b5c3e5 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/6ee13b1f362647948c98ec0833b5c3e5 2023-04-21 01:55:34,214 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 810791807ce3c9f7e9f14187121bf792/A of 810791807ce3c9f7e9f14187121bf792 into 6ee13b1f362647948c98ec0833b5c3e5(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:34,214 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:34,214 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., storeName=810791807ce3c9f7e9f14187121bf792/A, priority=13, startTime=1682042133286; duration=0sec 2023-04-21 01:55:34,214 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:34,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:34,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042194278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:34,279 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/e0335a30382f4ffbb6813416dbd3b248 2023-04-21 01:55:34,294 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/458efc37e1b948468492e9121e809d90 2023-04-21 01:55:34,298 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/c3d83ca586c3463f9ee666d0b8d1fcb8 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/c3d83ca586c3463f9ee666d0b8d1fcb8 2023-04-21 01:55:34,301 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/c3d83ca586c3463f9ee666d0b8d1fcb8, entries=150, sequenceid=240, filesize=25.3 K 2023-04-21 01:55:34,301 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/e0335a30382f4ffbb6813416dbd3b248 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/e0335a30382f4ffbb6813416dbd3b248 2023-04-21 01:55:34,305 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/e0335a30382f4ffbb6813416dbd3b248, entries=150, sequenceid=240, filesize=11.8 K 2023-04-21 01:55:34,305 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/458efc37e1b948468492e9121e809d90 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/458efc37e1b948468492e9121e809d90 2023-04-21 01:55:34,308 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/458efc37e1b948468492e9121e809d90, entries=150, sequenceid=240, filesize=11.8 K 2023-04-21 01:55:34,309 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 810791807ce3c9f7e9f14187121bf792 in 876ms, sequenceid=240, compaction requested=false 2023-04-21 01:55:34,309 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:34,309 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:34,309 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:34,309 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:34,309 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:34,309 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:34,309 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:34,311 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:34,311 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:34,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:34,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:34,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:34,311 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:34,311 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:34,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:34,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:34,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:34,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:34,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:34,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:34,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3735f90e[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:34,313 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:34,313 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:34,314 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:34,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:34,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:34,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:34,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:34,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:34,314 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:34,314 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:34,315 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:34,315 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:34,315 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:34,316 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:34,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:34,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:34,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:34,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:34,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:34,318 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:34,318 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:34,318 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:34,318 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:34,318 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:34,320 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:34,320 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:34,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:34,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:34,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:34,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:34,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:34,321 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:34,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:34,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:34,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:34,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:34,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:34,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:34,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:34,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:34,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:34,323 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:34,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:34,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:34,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:34,325 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:34,325 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:34,325 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:34,325 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:34,325 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:34,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:34,325 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:34,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:34,325 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:34,326 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:34,326 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:34,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:34,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:34,326 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:34,326 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:34,326 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:34,326 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:34,326 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:34,427 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:34,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:34,428 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:34,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:34,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:34,429 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:34,429 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:34,429 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:34,429 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:34,430 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:34,430 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:34,431 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:34,431 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:34,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:34,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:34,431 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:34,431 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:34,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:34,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:34,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:34,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:34,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:34,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:34,432 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:34,433 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:34,433 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:34,433 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:34,433 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:34,433 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:34,433 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:34,433 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 810791807ce3c9f7e9f14187121bf792 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-21 01:55:34,433 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=A 2023-04-21 01:55:34,433 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:34,433 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=B 2023-04-21 01:55:34,433 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:34,433 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=C 2023-04-21 01:55:34,433 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:34,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:34,587 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:34,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:34,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042194661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:34,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:34,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042194768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:34,852 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421cfae0cfa8eeb43d4b95117565c53e097 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421cfae0cfa8eeb43d4b95117565c53e097 2023-04-21 01:55:34,975 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:34,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042194973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:35,258 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=254, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/50409455b31544bb8948fb6fe42eccf6 2023-04-21 01:55:35,282 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:35,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042195279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:35,668 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/eb2595cfe58a4d8c85c53ead1819b366 2023-04-21 01:55:35,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:35,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042195787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,076 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/bee47c603b5d4989b31d032f992cecb8 2023-04-21 01:55:36,080 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/50409455b31544bb8948fb6fe42eccf6 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/50409455b31544bb8948fb6fe42eccf6 2023-04-21 01:55:36,084 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/50409455b31544bb8948fb6fe42eccf6, entries=150, sequenceid=254, filesize=25.3 K 2023-04-21 01:55:36,085 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/eb2595cfe58a4d8c85c53ead1819b366 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/eb2595cfe58a4d8c85c53ead1819b366 2023-04-21 01:55:36,089 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/eb2595cfe58a4d8c85c53ead1819b366, entries=150, sequenceid=254, filesize=11.8 K 2023-04-21 01:55:36,089 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/bee47c603b5d4989b31d032f992cecb8 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/bee47c603b5d4989b31d032f992cecb8 2023-04-21 01:55:36,093 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/bee47c603b5d4989b31d032f992cecb8, entries=150, sequenceid=254, filesize=11.8 K 2023-04-21 01:55:36,094 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 810791807ce3c9f7e9f14187121bf792 in 1661ms, sequenceid=254, compaction requested=true 2023-04-21 01:55:36,094 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:36,094 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:36,094 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:36,094 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:36,094 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,094 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:36,094 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:36,099 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,099 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:36,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:36,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:36,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:36,100 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:36,100 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:36,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:36,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:36,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:36,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:36,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b79bd37[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:36,101 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:36,101 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:36,102 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:36,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:36,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:36,103 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:36,103 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:36,103 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:36,103 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,103 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:36,104 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:36,104 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:36,104 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:36,104 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:36,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:36,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:36,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:36,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:36,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:36,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:36,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:36,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:36,109 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:36,109 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:36,109 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:36,109 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:36,109 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:36,118 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:36,118 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:36,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:36,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:36,118 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:36,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:36,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:36,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:36,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:36,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:36,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:36,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:36,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:36,119 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:36,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:36,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:36,120 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:36,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,124 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:36,124 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:36,124 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:36,124 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:36,124 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:36,124 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:36,124 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:36,124 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:36,124 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,124 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:36,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:36,124 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:36,124 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:36,124 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:36,125 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:36,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:36,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:36,225 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:36,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:36,229 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:36,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:36,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:36,233 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:36,233 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:36,233 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:36,233 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:36,233 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:36,233 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:36,234 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:36,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:36,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:36,234 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:36,235 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,235 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:36,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:36,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:36,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:36,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:36,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:36,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:36,240 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:36,240 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:36,240 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:36,244 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:36,244 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:36,244 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:36,244 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:36,244 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 810791807ce3c9f7e9f14187121bf792 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-21 01:55:36,244 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=A 2023-04-21 01:55:36,245 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:36,245 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=B 2023-04-21 01:55:36,245 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:36,245 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=C 2023-04-21 01:55:36,245 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:36,277 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304212ef1fb7229974ad98df8d2653e100d80 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304212ef1fb7229974ad98df8d2653e100d80 2023-04-21 01:55:36,295 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=273, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/6060b3d4602147de9800817951a89aac 2023-04-21 01:55:36,713 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/d8085a53d2fd472cae58af0af11f69af 2023-04-21 01:55:36,732 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/f9a1391d244e4c6cbdddae5e7915e663 2023-04-21 01:55:36,736 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/6060b3d4602147de9800817951a89aac as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/6060b3d4602147de9800817951a89aac 2023-04-21 01:55:36,745 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/6060b3d4602147de9800817951a89aac, entries=150, sequenceid=273, filesize=25.4 K 2023-04-21 01:55:36,746 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/d8085a53d2fd472cae58af0af11f69af as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/d8085a53d2fd472cae58af0af11f69af 2023-04-21 01:55:36,750 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/d8085a53d2fd472cae58af0af11f69af, entries=150, sequenceid=273, filesize=11.9 K 2023-04-21 01:55:36,751 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/f9a1391d244e4c6cbdddae5e7915e663 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/f9a1391d244e4c6cbdddae5e7915e663 2023-04-21 01:55:36,761 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/f9a1391d244e4c6cbdddae5e7915e663, entries=150, sequenceid=273, filesize=11.9 K 2023-04-21 01:55:36,762 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 810791807ce3c9f7e9f14187121bf792 in 518ms, sequenceid=273, compaction requested=true 2023-04-21 01:55:36,762 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:36,762 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:36,762 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:36,762 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:36,762 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,762 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:36,762 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:36,764 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,764 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:36,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:36,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:36,765 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:36,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:36,765 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:36,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:36,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:36,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:36,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:36,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@498331c9[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:36,767 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:36,767 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:36,768 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:36,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:36,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:36,768 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:36,768 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:36,768 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:36,768 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,768 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:36,769 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,769 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:36,770 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:36,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:36,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:36,770 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:36,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:36,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:36,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:36,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:36,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:36,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:36,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:36,772 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:36,772 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:36,773 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:36,773 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:36,773 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:36,776 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:36,776 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:36,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:36,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:36,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:36,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:36,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:36,776 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:36,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:36,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:36,776 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:36,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:36,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:36,778 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:36,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:36,780 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:36,780 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:36,780 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,780 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:36,780 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:36,780 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:36,780 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:36,780 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:36,780 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,780 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:36,780 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:36,780 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:36,780 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:36,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:36,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:36,780 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:36,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:36,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:36,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:36,881 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:36,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:36,884 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:36,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:36,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:36,885 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:36,886 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:36,886 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:36,886 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:36,886 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:36,886 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:36,887 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:36,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:36,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:36,888 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,888 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:36,888 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:36,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:36,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:36,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:36,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:36,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:36,890 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:36,890 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:36,890 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:36,890 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:36,890 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:36,891 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:36,891 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:36,891 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:36,892 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:36,892 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 810791807ce3c9f7e9f14187121bf792 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-21 01:55:36,892 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=A 2023-04-21 01:55:36,892 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:36,892 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=B 2023-04-21 01:55:36,892 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:36,892 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=C 2023-04-21 01:55:36,892 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:37,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:37,003 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:37,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042137064, max timestamp=1682042137095Num uniques 859; 2023-04-21 01:55:37,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042137064, max timestamp=1682042137095Num uniques 859; 2023-04-21 01:55:37,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:37,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042137064, max timestamp=1682042137095Num uniques 859; 2023-04-21 01:55:37,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:37,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:55:37,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682042137228, max timestamp=1682042137256Num uniques 1018; 2023-04-21 01:55:37,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682042137228, max timestamp=1682042137256Num uniques 1018; 2023-04-21 01:55:37,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-21 01:55:37,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682042137228, max timestamp=1682042137256Num uniques 1018; 2023-04-21 01:55:37,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-21 01:55:37,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-21 01:55:37,314 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304215c31b05b152948e1b49a70bffdd78e89 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304215c31b05b152948e1b49a70bffdd78e89 2023-04-21 01:55:37,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682042137355, max timestamp=1682042137383Num uniques 1027; 2023-04-21 01:55:37,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-21 01:55:37,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682042137355, max timestamp=1682042137383Num uniques 1027; 2023-04-21 01:55:37,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-21 01:55:37,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682042137355, max timestamp=1682042137383Num uniques 1027; 2023-04-21 01:55:37,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-21 01:55:37,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682042137462, max timestamp=1682042137476Num uniques 1036; 2023-04-21 01:55:37,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-21 01:55:37,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682042137462, max timestamp=1682042137476Num uniques 1036; 2023-04-21 01:55:37,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-21 01:55:37,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682042137462, max timestamp=1682042137476Num uniques 1036; 2023-04-21 01:55:37,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-21 01:55:37,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682042137547, max timestamp=1682042137586Num uniques 1045; 2023-04-21 01:55:37,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-21 01:55:37,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682042137547, max timestamp=1682042137586Num uniques 1045; 2023-04-21 01:55:37,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682042137547, max timestamp=1682042137586Num uniques 1045; 2023-04-21 01:55:37,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-21 01:55:37,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-21 01:55:37,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682042137685, max timestamp=1682042137714Num uniques 1104; 2023-04-21 01:55:37,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-21 01:55:37,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682042137685, max timestamp=1682042137714Num uniques 1104; 2023-04-21 01:55:37,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-21 01:55:37,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682042137685, max timestamp=1682042137714Num uniques 1104; 2023-04-21 01:55:37,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-21 01:55:37,736 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=277, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/efd68420f212494ea25283a91bd6a619 2023-04-21 01:55:37,754 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/4dc0c6b6237f4d779daa192599df4211 2023-04-21 01:55:37,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682042137815, max timestamp=1682042137851Num uniques 1013; 2023-04-21 01:55:37,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-21 01:55:37,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682042137815, max timestamp=1682042137851Num uniques 1013; 2023-04-21 01:55:37,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682042137815, max timestamp=1682042137851Num uniques 1013; 2023-04-21 01:55:37,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-21 01:55:37,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-21 01:55:37,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682042137951, max timestamp=1682042137986Num uniques 1022; 2023-04-21 01:55:37,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682042137951, max timestamp=1682042137986Num uniques 1022; 2023-04-21 01:55:37,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-21 01:55:37,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-21 01:55:37,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682042137951, max timestamp=1682042137986Num uniques 1022; 2023-04-21 01:55:37,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-21 01:55:38,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682042138041, max timestamp=1682042138117Num uniques 1031; 2023-04-21 01:55:38,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-21 01:55:38,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682042138041, max timestamp=1682042138117Num uniques 1031; 2023-04-21 01:55:38,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682042138041, max timestamp=1682042138117Num uniques 1031; 2023-04-21 01:55:38,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-21 01:55:38,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-21 01:55:38,174 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/e32fc5c896e94810ab8f23199da583d2 2023-04-21 01:55:38,177 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/efd68420f212494ea25283a91bd6a619 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/efd68420f212494ea25283a91bd6a619 2023-04-21 01:55:38,181 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/efd68420f212494ea25283a91bd6a619, entries=50, sequenceid=277, filesize=11.7 K 2023-04-21 01:55:38,182 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/4dc0c6b6237f4d779daa192599df4211 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/4dc0c6b6237f4d779daa192599df4211 2023-04-21 01:55:38,185 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/4dc0c6b6237f4d779daa192599df4211, entries=50, sequenceid=277, filesize=7.1 K 2023-04-21 01:55:38,185 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/e32fc5c896e94810ab8f23199da583d2 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/e32fc5c896e94810ab8f23199da583d2 2023-04-21 01:55:38,188 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/e32fc5c896e94810ab8f23199da583d2, entries=50, sequenceid=277, filesize=7.1 K 2023-04-21 01:55:38,189 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=87.22 KB/89310 for 810791807ce3c9f7e9f14187121bf792 in 1297ms, sequenceid=277, compaction requested=true 2023-04-21 01:55:38,189 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:38,189 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:38,189 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:38,189 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:38,189 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,189 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:38,189 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:38,191 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,191 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:38,191 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,191 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:38,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:38,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:38,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66664aa[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:38,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:38,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:38,193 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 810791807ce3c9f7e9f14187121bf792 3/3 column families, dataSize=93.93 KB heapSize=263.39 KB 2023-04-21 01:55:38,194 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=A 2023-04-21 01:55:38,194 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:38,194 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=B 2023-04-21 01:55:38,194 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:38,194 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=C 2023-04-21 01:55:38,194 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:38,195 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,195 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:38,196 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:38,196 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:38,196 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,196 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:38,197 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,197 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:38,197 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:38,197 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:38,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:38,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:38,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:38,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:38,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:38,200 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:38,200 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:38,200 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:38,200 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:38,201 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:38,202 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,202 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:38,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,203 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:38,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:38,203 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:38,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:38,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:38,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:38,205 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:38,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,207 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,207 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:38,207 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:38,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:38,207 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:38,207 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:38,207 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:38,207 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:38,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:38,207 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,207 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:38,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:38,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:38,208 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:38,207 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,208 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,208 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,208 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:38,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042198284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,308 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:38,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:38,310 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:38,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:38,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:38,311 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:38,311 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:38,311 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:38,311 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:38,312 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,312 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,314 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:38,314 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:38,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:38,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,314 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,314 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:38,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:38,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,315 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:38,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:38,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:38,315 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:38,315 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:38,315 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:38,316 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:38,316 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:38,316 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:38,316 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:38,316 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:38,316 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:38,316 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:38,316 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:38,316 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,316 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:38,316 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:38,317 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,317 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:38,318 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,318 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:38,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:38,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e8300e1[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:38,319 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:38,319 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,324 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,324 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:38,324 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:38,324 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:38,324 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,324 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:38,326 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,326 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:38,326 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:38,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:38,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:38,326 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:38,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:38,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:38,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:38,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:38,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:38,329 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:38,329 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:38,329 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:38,329 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:38,329 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:38,331 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,331 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,331 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:38,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:38,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:38,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:38,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:38,331 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:38,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:38,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,333 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:38,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,334 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,334 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,334 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,335 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,335 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:38,335 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:38,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:38,335 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:38,335 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:38,335 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,335 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:38,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:38,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:38,335 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:38,335 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:38,335 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:38,335 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:38,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:38,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 408 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042198396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,436 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:38,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:38,438 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:38,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:38,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:38,439 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:38,439 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:38,439 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:38,439 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:38,442 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,442 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,443 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:38,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:38,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:38,443 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,444 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,444 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:38,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:38,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:38,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:38,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:38,445 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:38,445 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:38,445 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:38,445 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:38,445 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:38,445 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:38,445 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:38,446 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:38,446 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:38,446 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:38,446 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:38,446 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,446 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:38,446 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:38,448 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,448 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:38,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:38,448 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,448 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:38,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:38,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:38,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:38,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:38,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18958603[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:38,450 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:38,450 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,451 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,451 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:38,451 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:38,451 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:38,452 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,452 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:38,454 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:38,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:38,454 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:38,454 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:38,454 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:38,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:38,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:38,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:38,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:38,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:38,458 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:38,459 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:38,459 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:38,459 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:38,459 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:38,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:38,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:38,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:38,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:38,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:38,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:38,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,461 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:38,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,465 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,468 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:38,468 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:38,468 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,468 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:38,468 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:38,468 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:38,468 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,468 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,468 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,468 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,468 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:38,468 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:38,468 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:38,468 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:38,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:38,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:38,569 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:38,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:38,571 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:38,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:38,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:38,575 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:38,575 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:38,575 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:38,575 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:38,576 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,576 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,577 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,577 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:38,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:38,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:38,577 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,577 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:38,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:38,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:38,585 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:38,585 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:38,585 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:38,586 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:38,586 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:38,597 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:38,597 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:38,597 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:38,606 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421cc25ec09ffa4498c9c6d7c5a3fcbd469 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421cc25ec09ffa4498c9c6d7c5a3fcbd469 2023-04-21 01:55:38,607 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:38,607 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:38,607 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:38,607 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:38,607 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:38,607 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,607 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:38,607 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:38,609 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:38,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:38,609 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:38,610 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,610 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:38,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:38,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:38,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:38,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4848f1a1[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:38,611 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:38,611 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,612 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,612 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:38,613 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,613 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:38,613 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:38,613 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:38,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:38,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 410 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042198606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,615 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,615 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:38,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:38,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:38,615 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:38,615 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:38,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:38,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:38,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:38,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:38,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:38,618 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:38,618 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:38,618 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:38,618 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:38,618 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:38,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:38,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:38,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:38,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:38,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:38,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:38,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,620 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:38,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,621 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:38,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,623 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,623 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,623 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:38,623 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:38,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:38,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:38,623 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,623 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:38,623 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:38,623 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:38,623 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,623 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:38,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:38,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:38,623 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:38,623 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,624 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,624 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,724 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:38,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:38,726 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:38,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:38,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:38,762 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:38,763 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:38,763 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:38,763 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:38,763 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,763 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,764 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,764 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:38,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:38,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:38,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,765 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,765 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:38,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:38,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,765 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:38,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:38,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:38,766 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:38,766 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:38,766 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:38,768 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:38,768 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:38,768 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:38,768 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:38,768 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:38,768 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:38,768 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:38,768 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:38,768 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,768 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:38,768 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:38,772 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:38,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:38,773 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:38,773 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,773 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:38,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:38,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:38,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:38,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dbe9f36[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:38,777 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:38,777 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,779 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,779 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,779 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:38,779 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:38,779 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:38,779 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:38,786 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,786 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:38,786 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:38,786 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:38,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:38,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:38,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:38,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:38,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:38,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:38,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:38,789 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:38,789 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:38,789 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:38,789 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:38,789 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:38,792 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,793 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:38,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:38,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,793 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:38,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:38,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:38,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:38,794 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:38,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,795 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:38,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,797 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:38,798 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:38,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:38,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:38,798 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,798 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:38,798 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:38,798 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:38,798 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:38,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:38,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:38,798 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,798 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:38,798 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,798 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,798 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,798 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:38,798 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,898 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:38,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:38,901 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:38,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:38,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:38,902 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:38,902 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:38,902 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:38,902 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:38,902 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,903 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,904 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:38,904 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:38,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:38,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,904 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,904 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:38,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:38,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:38,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:38,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:38,908 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:38,908 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:38,908 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:38,909 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:38,909 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:38,910 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:38,910 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:38,910 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:38,910 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:38,910 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:38,910 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:38,910 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,910 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:38,910 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:38,912 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,912 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:38,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:38,912 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,913 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:38,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:38,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:38,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:38,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:38,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bbc1c68[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:38,914 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:38,914 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,917 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,917 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:38,918 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:38,918 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:38,918 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,918 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:38,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:38,919 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:38,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 412 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042198916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,919 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:38,919 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:38,919 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:38,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:38,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:38,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:38,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:38,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:38,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:38,923 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:38,924 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:38,924 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:38,924 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:38,924 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:38,925 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,925 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:38,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,926 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:38,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:38,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:38,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:38,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,926 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:38,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:38,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,928 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:38,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:38,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,930 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,930 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,930 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:38,930 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:38,930 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:38,930 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:38,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:38,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:38,931 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:38,931 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:38,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:38,930 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:38,930 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,932 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:38,932 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:38,932 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:38,932 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:38,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:38,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:39,030 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=445, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/d23765ffa10a46b98681c220aff95966 2023-04-21 01:55:39,033 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:39,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:39,036 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:39,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:39,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:39,037 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:39,037 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:39,037 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:39,038 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:39,038 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,038 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:39,041 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,041 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:39,041 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,041 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:39,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:39,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:39,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:39,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:39,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:39,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:39,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:39,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:39,048 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:39,048 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:39,048 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:39,049 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:39,049 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:39,049 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:39,049 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:39,049 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:39,049 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:39,049 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:39,049 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:39,049 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,049 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:39,049 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:39,051 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,051 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,051 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:39,051 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:39,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:39,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:39,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:39,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:39,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:39,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:39,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fad3882[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:39,055 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:39,055 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,056 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:39,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:39,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:39,057 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,057 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:39,057 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:39,058 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:39,058 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:39,058 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:39,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:39,063 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:39,063 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:39,063 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:39,063 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:39,063 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:39,065 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,066 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,066 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:39,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:39,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:39,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:39,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:39,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:39,067 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:39,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:39,071 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,071 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:39,071 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:39,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:39,071 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:39,071 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,071 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,071 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:39,071 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,071 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,071 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:39,071 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:39,071 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:39,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:39,072 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:39,072 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:39,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:39,172 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:39,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:39,175 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:39,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:39,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:39,176 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:39,176 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:39,176 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:39,176 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:39,176 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,176 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:39,177 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,177 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:39,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:39,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:39,178 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,178 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:39,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:39,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:39,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:39,178 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:39,178 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:39,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:39,179 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:39,179 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:39,179 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:39,179 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:39,179 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:39,179 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:39,179 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:39,179 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:39,179 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:39,179 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:39,179 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:39,179 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,179 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:39,179 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:39,181 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,181 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:39,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:39,181 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,181 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:39,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:39,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:39,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:39,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:39,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:39,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40e70d2f[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:39,183 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:39,183 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,185 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,185 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:39,185 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:39,185 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:39,185 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,185 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:39,186 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,186 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:39,186 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:39,186 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:39,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:39,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:39,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:39,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:39,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:39,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:39,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:39,188 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:39,189 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:39,189 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:39,189 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:39,189 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:39,190 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,190 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:39,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,190 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:39,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:39,191 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:39,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:39,192 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,198 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,198 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:39,198 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:39,198 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:39,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:39,198 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:39,198 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:39,198 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:39,198 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,198 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:39,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:39,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:39,199 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:39,199 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:39,198 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,199 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,199 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:39,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:39,299 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:39,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:39,301 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:39,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:39,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:39,302 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:39,302 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:39,302 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:39,302 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:39,302 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,302 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:39,303 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,303 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:39,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:39,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:39,304 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,304 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:39,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:39,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:39,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:39,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:39,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:39,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:39,305 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:39,306 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:39,306 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:39,306 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:39,306 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:39,306 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:39,309 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,309 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:39,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:39,309 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,309 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:39,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:39,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:39,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:39,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:39,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49297d2a[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:39,310 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:39,310 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,311 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,311 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:39,311 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:39,312 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:39,312 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,312 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:39,313 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:39,313 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:39,313 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,313 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:39,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:39,315 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:39,315 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:39,315 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:39,315 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:39,315 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:39,321 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,321 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:39,321 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:39,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:39,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:39,323 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:39,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:39,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,327 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,327 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:39,327 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:39,327 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:39,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:39,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:39,327 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:39,327 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:39,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:39,327 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:39,327 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:39,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:39,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:39,327 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,327 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:39,327 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:39,327 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,327 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,328 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,427 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:39,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:39,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:39,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 414 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042199423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,430 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:39,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:39,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:39,431 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:39,431 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:39,431 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:39,431 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:39,431 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,431 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:39,432 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:39,432 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:39,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:39,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:39,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:39,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:39,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:39,434 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:39,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:39,434 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:39,434 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:39,434 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:39,434 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,434 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:39,434 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:39,434 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:39,434 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:39,434 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:39,435 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:39,435 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:39,435 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:39,435 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:39,435 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,435 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:39,435 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:39,438 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,438 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:39,438 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,438 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:39,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:39,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@526d773b[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:39,439 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:39,439 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,440 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,440 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:39,440 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:39,440 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:39,440 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,441 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:39,443 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:39,443 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,443 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:39,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:39,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:39,443 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:39,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:39,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:39,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:39,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:39,445 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:39,445 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:39,445 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:39,445 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:39,445 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:39,446 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,446 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,447 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:39,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:39,447 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,448 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:39,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:39,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:39,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:39,451 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:39,451 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:39,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:39,451 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:39,451 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:39,451 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:39,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:39,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:39,451 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:39,452 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:39,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:39,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:39,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:39,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:39,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:39,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:39,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,466 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=445 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/37c1eb8bc878439f81d1448d7ea0bb06 2023-04-21 01:55:39,486 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=445 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/a6fb80851aa24721b172a6b46cd0ef34 2023-04-21 01:55:39,489 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/d23765ffa10a46b98681c220aff95966 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/d23765ffa10a46b98681c220aff95966 2023-04-21 01:55:39,493 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/d23765ffa10a46b98681c220aff95966, entries=150, sequenceid=445, filesize=25.4 K 2023-04-21 01:55:39,494 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/37c1eb8bc878439f81d1448d7ea0bb06 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/37c1eb8bc878439f81d1448d7ea0bb06 2023-04-21 01:55:39,499 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/37c1eb8bc878439f81d1448d7ea0bb06, entries=150, sequenceid=445, filesize=11.9 K 2023-04-21 01:55:39,500 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/a6fb80851aa24721b172a6b46cd0ef34 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/a6fb80851aa24721b172a6b46cd0ef34 2023-04-21 01:55:39,505 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/a6fb80851aa24721b172a6b46cd0ef34, entries=150, sequenceid=445, filesize=11.9 K 2023-04-21 01:55:39,506 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~263.34 KB/269664, currentSize=87.22 KB/89310 for 810791807ce3c9f7e9f14187121bf792 in 1313ms, sequenceid=445, compaction requested=true 2023-04-21 01:55:39,506 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:39,506 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:39,506 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-21 01:55:39,506 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-21 01:55:39,506 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:39,506 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:39,509 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 68292 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-21 01:55:39,509 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 810791807ce3c9f7e9f14187121bf792/B is initiating minor compaction (all files) 2023-04-21 01:55:39,509 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 810791807ce3c9f7e9f14187121bf792/B in TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:39,509 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/8c24803137694d338a2a4e94491c7285, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/e0335a30382f4ffbb6813416dbd3b248, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/eb2595cfe58a4d8c85c53ead1819b366, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/d8085a53d2fd472cae58af0af11f69af, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/4dc0c6b6237f4d779daa192599df4211, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/37c1eb8bc878439f81d1448d7ea0bb06] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp, totalSize=66.7 K 2023-04-21 01:55:39,510 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 8c24803137694d338a2a4e94491c7285, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682042130038 2023-04-21 01:55:39,510 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 142248 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-21 01:55:39,511 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting e0335a30382f4ffbb6813416dbd3b248, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1682042131755 2023-04-21 01:55:39,511 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 810791807ce3c9f7e9f14187121bf792/A is initiating minor compaction (all files) 2023-04-21 01:55:39,511 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 810791807ce3c9f7e9f14187121bf792/A in TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:39,511 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/6ee13b1f362647948c98ec0833b5c3e5, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/c3d83ca586c3463f9ee666d0b8d1fcb8, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/50409455b31544bb8948fb6fe42eccf6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/6060b3d4602147de9800817951a89aac, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/efd68420f212494ea25283a91bd6a619, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/d23765ffa10a46b98681c220aff95966] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp, totalSize=138.9 K 2023-04-21 01:55:39,512 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting eb2595cfe58a4d8c85c53ead1819b366, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1682042133899 2023-04-21 01:55:39,513 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 6ee13b1f362647948c98ec0833b5c3e5, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682042130038 2023-04-21 01:55:39,513 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting d8085a53d2fd472cae58af0af11f69af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1682042134640 2023-04-21 01:55:39,513 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 4dc0c6b6237f4d779daa192599df4211, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1682042136842 2023-04-21 01:55:39,513 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting c3d83ca586c3463f9ee666d0b8d1fcb8, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1682042131755 2023-04-21 01:55:39,514 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 37c1eb8bc878439f81d1448d7ea0bb06, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1682042138041 2023-04-21 01:55:39,515 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 50409455b31544bb8948fb6fe42eccf6, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1682042133899 2023-04-21 01:55:39,515 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 6060b3d4602147de9800817951a89aac, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1682042134640 2023-04-21 01:55:39,516 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting efd68420f212494ea25283a91bd6a619, keycount=50, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1682042136842 2023-04-21 01:55:39,516 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting d23765ffa10a46b98681c220aff95966, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1682042138041 2023-04-21 01:55:39,526 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 810791807ce3c9f7e9f14187121bf792#B#compaction#396 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:39,544 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 810791807ce3c9f7e9f14187121bf792#A#compaction#397 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:39,552 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:39,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:39,555 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:39,557 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/07e8b394c2c74bacb060ce28586f77ae as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/07e8b394c2c74bacb060ce28586f77ae 2023-04-21 01:55:39,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:39,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:39,560 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:39,560 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:39,560 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:39,560 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:39,560 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,561 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:39,562 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 810791807ce3c9f7e9f14187121bf792/B of 810791807ce3c9f7e9f14187121bf792 into 07e8b394c2c74bacb060ce28586f77ae(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:39,562 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:39,562 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., storeName=810791807ce3c9f7e9f14187121bf792/B, priority=10, startTime=1682042139506; duration=0sec 2023-04-21 01:55:39,562 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:39,562 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-21 01:55:39,563 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,563 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:39,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:39,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:39,563 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:39,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:39,563 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:39,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:39,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:39,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:39,565 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 68292 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-21 01:55:39,565 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 810791807ce3c9f7e9f14187121bf792/C is initiating minor compaction (all files) 2023-04-21 01:55:39,565 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 810791807ce3c9f7e9f14187121bf792/C in TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:39,565 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/559258a7590e4fffad960d82927d6455, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/458efc37e1b948468492e9121e809d90, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/bee47c603b5d4989b31d032f992cecb8, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/f9a1391d244e4c6cbdddae5e7915e663, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/e32fc5c896e94810ab8f23199da583d2, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/a6fb80851aa24721b172a6b46cd0ef34] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp, totalSize=66.7 K 2023-04-21 01:55:39,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:39,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:39,568 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:39,568 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:39,568 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:39,568 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 559258a7590e4fffad960d82927d6455, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682042130038 2023-04-21 01:55:39,569 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 458efc37e1b948468492e9121e809d90, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1682042131755 2023-04-21 01:55:39,569 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting bee47c603b5d4989b31d032f992cecb8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1682042133899 2023-04-21 01:55:39,570 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting f9a1391d244e4c6cbdddae5e7915e663, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1682042134640 2023-04-21 01:55:39,570 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting e32fc5c896e94810ab8f23199da583d2, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1682042136842 2023-04-21 01:55:39,570 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting a6fb80851aa24721b172a6b46cd0ef34, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1682042138041 2023-04-21 01:55:39,572 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:39,572 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:39,572 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:39,572 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:39,572 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 810791807ce3c9f7e9f14187121bf792 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-21 01:55:39,572 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=A 2023-04-21 01:55:39,572 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:39,572 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=B 2023-04-21 01:55:39,573 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:39,573 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=C 2023-04-21 01:55:39,573 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:39,583 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 810791807ce3c9f7e9f14187121bf792#C#compaction#398 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:39,594 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/0c4e668a462e4adaa799416a2e05e8c3 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/0c4e668a462e4adaa799416a2e05e8c3 2023-04-21 01:55:39,599 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 810791807ce3c9f7e9f14187121bf792/A of 810791807ce3c9f7e9f14187121bf792 into 0c4e668a462e4adaa799416a2e05e8c3(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:39,600 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:39,600 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., storeName=810791807ce3c9f7e9f14187121bf792/A, priority=10, startTime=1682042139506; duration=0sec 2023-04-21 01:55:39,600 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:39,612 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/26b3a5f2ad164c95a3404db4aa0155c9 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/26b3a5f2ad164c95a3404db4aa0155c9 2023-04-21 01:55:39,616 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 810791807ce3c9f7e9f14187121bf792/C of 810791807ce3c9f7e9f14187121bf792 into 26b3a5f2ad164c95a3404db4aa0155c9(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:39,616 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:39,616 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., storeName=810791807ce3c9f7e9f14187121bf792/C, priority=10, startTime=1682042139506; duration=0sec 2023-04-21 01:55:39,616 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:40,010 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304211539a178fce845e9979418651fd41de6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304211539a178fce845e9979418651fd41de6 2023-04-21 01:55:40,017 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=462, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/427a9d8f015741ee8bcd7f021f9b62ae 2023-04-21 01:55:40,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:40,439 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:40,440 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=462 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/27d2a0f2fee642ddb2c683ee48eab658 2023-04-21 01:55:40,483 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:40,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 429 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042200479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:40,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:40,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 431 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042200584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:40,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:40,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 433 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042200787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:40,800 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-21 01:55:40,855 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/711650f3b39d464dbef6159ea38056f7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/ca0087d054e5461585e3ecdb3ee62066, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/5822b1bd8aa048149b42825e8dd661b3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/13aae40fa5b640309f9cc9570d01607b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/6ea033cb25d64750ac9f63bd6e7e45dc, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/376e2d48db9548f8b5f5d1a7a2232e7a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/551df64958b94bb5b4d2a214ab89510e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/7d3cd9326f194cc79647a55b07418acf, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/1c3eb3bc88ce460c96db5a08cb8e6ab7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/4213cd5e78e14a5695a52d2231cfa5eb, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/2729aeaafd3e435384e3d550d7eed8d3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/caf6e2967e984825b79fbf58003128d5, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/0c0a443bf26e4091bbdc8c4fd94feb6c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/c7ef206510e041048387afb007bcd09f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/6ee13b1f362647948c98ec0833b5c3e5, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/3a7016e12ad146bdb49ddb05729e8297, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/c3d83ca586c3463f9ee666d0b8d1fcb8, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/50409455b31544bb8948fb6fe42eccf6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/6060b3d4602147de9800817951a89aac, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/efd68420f212494ea25283a91bd6a619, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/d23765ffa10a46b98681c220aff95966] to archive 2023-04-21 01:55:40,856 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-21 01:55:40,857 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=462 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/379abf706ff4411eb29e6ee0148d8cd6 2023-04-21 01:55:40,870 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/711650f3b39d464dbef6159ea38056f7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/711650f3b39d464dbef6159ea38056f7 2023-04-21 01:55:40,871 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/ca0087d054e5461585e3ecdb3ee62066 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/ca0087d054e5461585e3ecdb3ee62066 2023-04-21 01:55:40,873 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/5822b1bd8aa048149b42825e8dd661b3 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/5822b1bd8aa048149b42825e8dd661b3 2023-04-21 01:55:40,875 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/13aae40fa5b640309f9cc9570d01607b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/13aae40fa5b640309f9cc9570d01607b 2023-04-21 01:55:40,876 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/6ea033cb25d64750ac9f63bd6e7e45dc to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/6ea033cb25d64750ac9f63bd6e7e45dc 2023-04-21 01:55:40,876 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/427a9d8f015741ee8bcd7f021f9b62ae as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/427a9d8f015741ee8bcd7f021f9b62ae 2023-04-21 01:55:40,878 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/376e2d48db9548f8b5f5d1a7a2232e7a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/376e2d48db9548f8b5f5d1a7a2232e7a 2023-04-21 01:55:40,881 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/551df64958b94bb5b4d2a214ab89510e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/551df64958b94bb5b4d2a214ab89510e 2023-04-21 01:55:40,882 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/7d3cd9326f194cc79647a55b07418acf to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/7d3cd9326f194cc79647a55b07418acf 2023-04-21 01:55:40,884 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/1c3eb3bc88ce460c96db5a08cb8e6ab7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/1c3eb3bc88ce460c96db5a08cb8e6ab7 2023-04-21 01:55:40,886 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/4213cd5e78e14a5695a52d2231cfa5eb to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/4213cd5e78e14a5695a52d2231cfa5eb 2023-04-21 01:55:40,888 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/2729aeaafd3e435384e3d550d7eed8d3 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/2729aeaafd3e435384e3d550d7eed8d3 2023-04-21 01:55:40,891 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/caf6e2967e984825b79fbf58003128d5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/caf6e2967e984825b79fbf58003128d5 2023-04-21 01:55:40,892 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/0c0a443bf26e4091bbdc8c4fd94feb6c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/0c0a443bf26e4091bbdc8c4fd94feb6c 2023-04-21 01:55:40,895 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/c7ef206510e041048387afb007bcd09f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/c7ef206510e041048387afb007bcd09f 2023-04-21 01:55:40,897 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/6ee13b1f362647948c98ec0833b5c3e5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/6ee13b1f362647948c98ec0833b5c3e5 2023-04-21 01:55:40,898 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/3a7016e12ad146bdb49ddb05729e8297 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/3a7016e12ad146bdb49ddb05729e8297 2023-04-21 01:55:40,901 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/c3d83ca586c3463f9ee666d0b8d1fcb8 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/c3d83ca586c3463f9ee666d0b8d1fcb8 2023-04-21 01:55:40,902 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/50409455b31544bb8948fb6fe42eccf6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/50409455b31544bb8948fb6fe42eccf6 2023-04-21 01:55:40,904 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/6060b3d4602147de9800817951a89aac to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/6060b3d4602147de9800817951a89aac 2023-04-21 01:55:40,906 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/efd68420f212494ea25283a91bd6a619 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/efd68420f212494ea25283a91bd6a619 2023-04-21 01:55:40,908 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/d23765ffa10a46b98681c220aff95966 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/d23765ffa10a46b98681c220aff95966 2023-04-21 01:55:40,915 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/427a9d8f015741ee8bcd7f021f9b62ae, entries=150, sequenceid=462, filesize=25.4 K 2023-04-21 01:55:40,916 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/27d2a0f2fee642ddb2c683ee48eab658 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/27d2a0f2fee642ddb2c683ee48eab658 2023-04-21 01:55:40,916 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/7d45932cf5f84010be3a9e47ac43a292, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/7b5d376f693e4f7291382b161d7b44ca, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/8385b6720d924881a800ee7ec842b097, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/0fe479e55fac449d8c3b3cbb8ffc833d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/eae686432a0241a98c4ab77abd1ed998, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/7450abb6ed034cc0924c6a91ce9ae8a1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/c5a0465631e9403a806495fd5d40fb58, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/5f02dbae1d4e45d0859ba6e5dbf69b4d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/ef0eeabe9a7948e0a16b18df508cce51, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/27c87c2b63804b78808c4150d02f0983, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/71caafa4f887495f87a52dad3060c1f0, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/4bd2840d302549fa8953c2c154ef2182, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/4001f5ae3f0d4dba99f7a526a52efc6a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/8f95991f4a044166baf749083b25b26d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/8c24803137694d338a2a4e94491c7285, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/1ecdd21062c04250a62dff078558a0c7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/e0335a30382f4ffbb6813416dbd3b248, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/eb2595cfe58a4d8c85c53ead1819b366, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/d8085a53d2fd472cae58af0af11f69af, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/4dc0c6b6237f4d779daa192599df4211, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/37c1eb8bc878439f81d1448d7ea0bb06] to archive 2023-04-21 01:55:40,917 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-21 01:55:40,921 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/7d45932cf5f84010be3a9e47ac43a292 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/7d45932cf5f84010be3a9e47ac43a292 2023-04-21 01:55:40,923 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/7b5d376f693e4f7291382b161d7b44ca to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/7b5d376f693e4f7291382b161d7b44ca 2023-04-21 01:55:40,926 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/8385b6720d924881a800ee7ec842b097 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/8385b6720d924881a800ee7ec842b097 2023-04-21 01:55:40,931 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/0fe479e55fac449d8c3b3cbb8ffc833d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/0fe479e55fac449d8c3b3cbb8ffc833d 2023-04-21 01:55:40,933 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/eae686432a0241a98c4ab77abd1ed998 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/eae686432a0241a98c4ab77abd1ed998 2023-04-21 01:55:40,935 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/7450abb6ed034cc0924c6a91ce9ae8a1 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/7450abb6ed034cc0924c6a91ce9ae8a1 2023-04-21 01:55:40,936 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/27d2a0f2fee642ddb2c683ee48eab658, entries=150, sequenceid=462, filesize=11.9 K 2023-04-21 01:55:40,937 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/c5a0465631e9403a806495fd5d40fb58 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/c5a0465631e9403a806495fd5d40fb58 2023-04-21 01:55:40,937 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/379abf706ff4411eb29e6ee0148d8cd6 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/379abf706ff4411eb29e6ee0148d8cd6 2023-04-21 01:55:40,940 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/5f02dbae1d4e45d0859ba6e5dbf69b4d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/5f02dbae1d4e45d0859ba6e5dbf69b4d 2023-04-21 01:55:40,947 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/ef0eeabe9a7948e0a16b18df508cce51 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/ef0eeabe9a7948e0a16b18df508cce51 2023-04-21 01:55:40,948 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/27c87c2b63804b78808c4150d02f0983 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/27c87c2b63804b78808c4150d02f0983 2023-04-21 01:55:40,949 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/379abf706ff4411eb29e6ee0148d8cd6, entries=150, sequenceid=462, filesize=11.9 K 2023-04-21 01:55:40,949 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 810791807ce3c9f7e9f14187121bf792 in 1377ms, sequenceid=462, compaction requested=false 2023-04-21 01:55:40,950 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:40,950 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:40,950 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:40,950 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:40,950 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:40,950 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:40,950 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:40,950 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/71caafa4f887495f87a52dad3060c1f0 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/71caafa4f887495f87a52dad3060c1f0 2023-04-21 01:55:40,951 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/4bd2840d302549fa8953c2c154ef2182 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/4bd2840d302549fa8953c2c154ef2182 2023-04-21 01:55:40,952 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:40,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:40,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:40,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:40,952 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:40,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:40,953 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:40,953 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:40,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:40,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:40,953 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/4001f5ae3f0d4dba99f7a526a52efc6a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/4001f5ae3f0d4dba99f7a526a52efc6a 2023-04-21 01:55:40,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:40,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:40,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:40,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74223cdc[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:40,954 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:40,954 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:40,955 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/8f95991f4a044166baf749083b25b26d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/8f95991f4a044166baf749083b25b26d 2023-04-21 01:55:40,956 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:40,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:40,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:40,956 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:40,956 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:40,956 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:40,956 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:40,956 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/8c24803137694d338a2a4e94491c7285 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/8c24803137694d338a2a4e94491c7285 2023-04-21 01:55:40,956 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:40,958 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:40,958 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:40,958 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:40,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:40,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:40,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:40,958 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:40,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:40,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:40,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:40,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:40,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:40,960 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/1ecdd21062c04250a62dff078558a0c7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/1ecdd21062c04250a62dff078558a0c7 2023-04-21 01:55:40,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:40,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:40,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:40,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:40,961 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:40,961 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/e0335a30382f4ffbb6813416dbd3b248 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/e0335a30382f4ffbb6813416dbd3b248 2023-04-21 01:55:40,961 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:40,962 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:40,962 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:40,962 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:40,964 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/eb2595cfe58a4d8c85c53ead1819b366 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/eb2595cfe58a4d8c85c53ead1819b366 2023-04-21 01:55:40,968 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:40,968 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:40,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:40,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:40,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:40,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:40,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:40,969 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:40,969 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:40,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:40,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:40,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:40,970 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:40,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:40,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:40,971 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/d8085a53d2fd472cae58af0af11f69af to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/d8085a53d2fd472cae58af0af11f69af 2023-04-21 01:55:40,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:40,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:40,973 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/4dc0c6b6237f4d779daa192599df4211 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/4dc0c6b6237f4d779daa192599df4211 2023-04-21 01:55:40,975 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/37c1eb8bc878439f81d1448d7ea0bb06 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/37c1eb8bc878439f81d1448d7ea0bb06 2023-04-21 01:55:40,978 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:40,978 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:40,978 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:40,978 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:40,978 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:40,978 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:40,978 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:40,978 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:40,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:40,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:40,978 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:40,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:40,978 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:40,978 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:40,978 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:40,978 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:40,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:40,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:40,979 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:40,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:40,980 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/1c3d9293f6884bf4b3dafd86afddc580, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/fac8d20d26714c0faea8821df7c92c19, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/ebba7854f9684b5b90ac5837dd8d67ec, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/06d3668f656e4689a8a5b5bede6b2a27, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/c02ff544d6c44698b8d7ed87a6c18bc9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/45c5728448024b9b96ba0ebedd40abdf, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/cbc01504270c4a8ab34fd780c87b9354, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/44f43da7d4ed48e1b6c3f2225ca7b036, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/709547f0c103403abdbbdda3348dbc64, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/1e1ac9d19e30442c914e1d9754343be3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/23444fbbcd444b2eb310e4d80274bc6d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/5fa5a8a1b9004c518d23ba5b7dfc6d55, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/7968bac842a14e69850dd866d4df124a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/3afd20756c8d49d9b1a1b159db16a627, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/559258a7590e4fffad960d82927d6455, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/96fb6f99f75d46c99973163a70a48637, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/458efc37e1b948468492e9121e809d90, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/bee47c603b5d4989b31d032f992cecb8, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/f9a1391d244e4c6cbdddae5e7915e663, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/e32fc5c896e94810ab8f23199da583d2, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/a6fb80851aa24721b172a6b46cd0ef34] to archive 2023-04-21 01:55:40,981 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-21 01:55:40,982 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/1c3d9293f6884bf4b3dafd86afddc580 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/1c3d9293f6884bf4b3dafd86afddc580 2023-04-21 01:55:40,984 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/fac8d20d26714c0faea8821df7c92c19 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/fac8d20d26714c0faea8821df7c92c19 2023-04-21 01:55:40,989 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/ebba7854f9684b5b90ac5837dd8d67ec to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/ebba7854f9684b5b90ac5837dd8d67ec 2023-04-21 01:55:40,990 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/06d3668f656e4689a8a5b5bede6b2a27 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/06d3668f656e4689a8a5b5bede6b2a27 2023-04-21 01:55:40,992 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/c02ff544d6c44698b8d7ed87a6c18bc9 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/c02ff544d6c44698b8d7ed87a6c18bc9 2023-04-21 01:55:40,994 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/45c5728448024b9b96ba0ebedd40abdf to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/45c5728448024b9b96ba0ebedd40abdf 2023-04-21 01:55:40,995 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/cbc01504270c4a8ab34fd780c87b9354 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/cbc01504270c4a8ab34fd780c87b9354 2023-04-21 01:55:40,996 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/44f43da7d4ed48e1b6c3f2225ca7b036 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/44f43da7d4ed48e1b6c3f2225ca7b036 2023-04-21 01:55:41,008 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/709547f0c103403abdbbdda3348dbc64 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/709547f0c103403abdbbdda3348dbc64 2023-04-21 01:55:41,011 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/1e1ac9d19e30442c914e1d9754343be3 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/1e1ac9d19e30442c914e1d9754343be3 2023-04-21 01:55:41,014 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/23444fbbcd444b2eb310e4d80274bc6d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/23444fbbcd444b2eb310e4d80274bc6d 2023-04-21 01:55:41,015 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/5fa5a8a1b9004c518d23ba5b7dfc6d55 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/5fa5a8a1b9004c518d23ba5b7dfc6d55 2023-04-21 01:55:41,016 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/7968bac842a14e69850dd866d4df124a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/7968bac842a14e69850dd866d4df124a 2023-04-21 01:55:41,018 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/3afd20756c8d49d9b1a1b159db16a627 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/3afd20756c8d49d9b1a1b159db16a627 2023-04-21 01:55:41,020 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/559258a7590e4fffad960d82927d6455 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/559258a7590e4fffad960d82927d6455 2023-04-21 01:55:41,021 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/96fb6f99f75d46c99973163a70a48637 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/96fb6f99f75d46c99973163a70a48637 2023-04-21 01:55:41,023 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/458efc37e1b948468492e9121e809d90 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/458efc37e1b948468492e9121e809d90 2023-04-21 01:55:41,025 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/bee47c603b5d4989b31d032f992cecb8 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/bee47c603b5d4989b31d032f992cecb8 2023-04-21 01:55:41,028 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/f9a1391d244e4c6cbdddae5e7915e663 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/f9a1391d244e4c6cbdddae5e7915e663 2023-04-21 01:55:41,029 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/e32fc5c896e94810ab8f23199da583d2 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/e32fc5c896e94810ab8f23199da583d2 2023-04-21 01:55:41,030 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/a6fb80851aa24721b172a6b46cd0ef34 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/a6fb80851aa24721b172a6b46cd0ef34 2023-04-21 01:55:41,079 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:41,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:41,081 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:41,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:41,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:41,083 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:41,083 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:41,083 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:41,083 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:41,083 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:41,084 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:41,085 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:41,085 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:41,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:41,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:41,086 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:41,086 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:41,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:41,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:41,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:41,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:41,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:41,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:41,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:41,087 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:41,087 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:41,087 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:41,087 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:41,087 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:41,087 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:41,087 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. started... 2023-04-21 01:55:41,087 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 810791807ce3c9f7e9f14187121bf792 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-21 01:55:41,088 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=A 2023-04-21 01:55:41,088 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:41,088 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=B 2023-04-21 01:55:41,088 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:41,088 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=C 2023-04-21 01:55:41,088 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:41,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:41,098 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. as already flushing 2023-04-21 01:55:41,119 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304216704a3a89c9a4ecbb9242e8da82772a4 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304216704a3a89c9a4ecbb9242e8da82772a4 2023-04-21 01:55:41,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:41,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 448 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042201174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:41,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:41,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 450 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042201278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:41,397 DEBUG [Thread-2540] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x68a0e21a to 127.0.0.1:58383 2023-04-21 01:55:41,397 DEBUG [Thread-2540] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:55:41,397 DEBUG [Thread-2536] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e01d5e5 to 127.0.0.1:58383 2023-04-21 01:55:41,397 DEBUG [Thread-2536] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:55:41,397 DEBUG [Thread-2542] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ec12b02 to 127.0.0.1:58383 2023-04-21 01:55:41,397 DEBUG [Thread-2542] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:55:41,402 DEBUG [Thread-2544] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d7ba242 to 127.0.0.1:58383 2023-04-21 01:55:41,402 DEBUG [Thread-2544] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:55:41,406 DEBUG [Thread-2538] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51ccd8b7 to 127.0.0.1:58383 2023-04-21 01:55:41,406 DEBUG [Thread-2538] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:55:41,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:41,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 452 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042201486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:41,542 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=480, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/3942af1dbd594db58c8b4cd4696b33b3 2023-04-21 01:55:41,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:41,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 454 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38716 deadline: 1682042201788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:41,953 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/b45516b9e6f34bbbbbd9484ab90d9794 2023-04-21 01:55:41,963 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/d0e078b59a0a4c2d9c7fa2eea2c3b42b 2023-04-21 01:55:41,969 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/3942af1dbd594db58c8b4cd4696b33b3 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/3942af1dbd594db58c8b4cd4696b33b3 2023-04-21 01:55:41,976 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/3942af1dbd594db58c8b4cd4696b33b3, entries=150, sequenceid=480, filesize=25.4 K 2023-04-21 01:55:41,976 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/b45516b9e6f34bbbbbd9484ab90d9794 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/b45516b9e6f34bbbbbd9484ab90d9794 2023-04-21 01:55:41,980 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/b45516b9e6f34bbbbbd9484ab90d9794, entries=150, sequenceid=480, filesize=11.9 K 2023-04-21 01:55:41,980 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/d0e078b59a0a4c2d9c7fa2eea2c3b42b as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/d0e078b59a0a4c2d9c7fa2eea2c3b42b 2023-04-21 01:55:41,983 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/d0e078b59a0a4c2d9c7fa2eea2c3b42b, entries=150, sequenceid=480, filesize=11.9 K 2023-04-21 01:55:41,984 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 810791807ce3c9f7e9f14187121bf792 in 897ms, sequenceid=480, compaction requested=true 2023-04-21 01:55:41,984 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:41,984 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:41,984 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:41,986 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:41,986 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:41,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:41,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:41,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:41,986 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:41,986 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:41,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:41,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:41,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:41,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:41,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:41,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:41,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c54c902[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:41,987 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:41,988 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:41,990 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:41,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:41,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:41,990 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:41,990 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:41,990 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:41,990 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:41,990 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:41,991 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:41,992 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:41,992 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:41,992 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:41,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:41,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:41,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:41,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:41,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:41,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:41,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:41,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:41,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:41,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:41,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:41,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:41,994 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:41,994 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:41,994 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:41,994 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:41,994 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:41,996 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:41,996 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:41,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:41,996 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:41,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:41,996 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:41,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:41,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:41,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:41,997 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:41,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:41,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:41,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:41,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:42,000 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:42,000 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:42,000 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:42,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:42,000 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:42,000 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:42,000 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:42,000 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:42,000 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:42,000 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:42,000 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:42,000 DEBUG [Thread-2535] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:42,001 DEBUG [Thread-2535] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:42,001 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:42,001 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:42,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:42,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:42,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:42,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:42,101 DEBUG [Thread-2535] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:42,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:42,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:42,292 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 810791807ce3c9f7e9f14187121bf792 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-21 01:55:42,292 DEBUG [Thread-2531] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x71b5058b to 127.0.0.1:58383 2023-04-21 01:55:42,292 DEBUG [Thread-2531] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:55:42,292 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=A 2023-04-21 01:55:42,292 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:42,292 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=B 2023-04-21 01:55:42,292 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:42,292 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=C 2023-04-21 01:55:42,292 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:42,705 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304215c1c5658e03e46a796e151ebad64e9ba to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304215c1c5658e03e46a796e151ebad64e9ba 2023-04-21 01:55:43,088 DEBUG [Thread-2529] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38341134 to 127.0.0.1:58383 2023-04-21 01:55:43,088 DEBUG [Thread-2529] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:55:43,111 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=497, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/9176f1825be54348862e2e6f4edbd0e7 2023-04-21 01:55:43,116 DEBUG [Thread-2533] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3620ee35 to 127.0.0.1:58383 2023-04-21 01:55:43,116 DEBUG [Thread-2533] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:55:43,120 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/eac52abfadc7445f9d31f2a1b170be37 2023-04-21 01:55:43,128 DEBUG [Thread-2527] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bcfc1e0 to 127.0.0.1:58383 2023-04-21 01:55:43,128 DEBUG [Thread-2527] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:55:43,130 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/94639410d832481cad9507b3fde09f92 2023-04-21 01:55:43,133 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/9176f1825be54348862e2e6f4edbd0e7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/9176f1825be54348862e2e6f4edbd0e7 2023-04-21 01:55:43,135 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/9176f1825be54348862e2e6f4edbd0e7, entries=150, sequenceid=497, filesize=25.4 K 2023-04-21 01:55:43,136 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/eac52abfadc7445f9d31f2a1b170be37 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/eac52abfadc7445f9d31f2a1b170be37 2023-04-21 01:55:43,142 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/eac52abfadc7445f9d31f2a1b170be37, entries=150, sequenceid=497, filesize=11.9 K 2023-04-21 01:55:43,142 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/94639410d832481cad9507b3fde09f92 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/94639410d832481cad9507b3fde09f92 2023-04-21 01:55:43,145 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/94639410d832481cad9507b3fde09f92, entries=150, sequenceid=497, filesize=11.9 K 2023-04-21 01:55:43,145 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=20.13 KB/20610 for 810791807ce3c9f7e9f14187121bf792 in 853ms, sequenceid=497, compaction requested=true 2023-04-21 01:55:43,145 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:43,146 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:43,146 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:43,146 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:55:43,146 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:55:43,146 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:43,147 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:55:43,147 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 810791807ce3c9f7e9f14187121bf792/A is initiating minor compaction (all files) 2023-04-21 01:55:43,147 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 810791807ce3c9f7e9f14187121bf792/A in TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:43,147 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/0c4e668a462e4adaa799416a2e05e8c3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/427a9d8f015741ee8bcd7f021f9b62ae, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/3942af1dbd594db58c8b4cd4696b33b3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/9176f1825be54348862e2e6f4edbd0e7] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp, totalSize=102.4 K 2023-04-21 01:55:43,147 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:55:43,147 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 810791807ce3c9f7e9f14187121bf792/B is initiating minor compaction (all files) 2023-04-21 01:55:43,147 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 0c4e668a462e4adaa799416a2e05e8c3, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1682042138041 2023-04-21 01:55:43,147 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 810791807ce3c9f7e9f14187121bf792/B in TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:43,148 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/07e8b394c2c74bacb060ce28586f77ae, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/27d2a0f2fee642ddb2c683ee48eab658, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/b45516b9e6f34bbbbbd9484ab90d9794, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/eac52abfadc7445f9d31f2a1b170be37] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp, totalSize=48.3 K 2023-04-21 01:55:43,148 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 427a9d8f015741ee8bcd7f021f9b62ae, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=462, earliestPutTs=1682042138261 2023-04-21 01:55:43,148 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 07e8b394c2c74bacb060ce28586f77ae, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1682042138041 2023-04-21 01:55:43,148 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 3942af1dbd594db58c8b4cd4696b33b3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1682042140470 2023-04-21 01:55:43,148 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 27d2a0f2fee642ddb2c683ee48eab658, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=462, earliestPutTs=1682042138261 2023-04-21 01:55:43,148 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 9176f1825be54348862e2e6f4edbd0e7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1682042141165 2023-04-21 01:55:43,149 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting b45516b9e6f34bbbbbd9484ab90d9794, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1682042140470 2023-04-21 01:55:43,149 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting eac52abfadc7445f9d31f2a1b170be37, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1682042141165 2023-04-21 01:55:43,158 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 810791807ce3c9f7e9f14187121bf792#B#compaction#409 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:43,169 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 810791807ce3c9f7e9f14187121bf792#A#compaction#408 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:43,174 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/4fc5447597c04b56876f663cc270765e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/4fc5447597c04b56876f663cc270765e 2023-04-21 01:55:43,178 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 810791807ce3c9f7e9f14187121bf792/B of 810791807ce3c9f7e9f14187121bf792 into 4fc5447597c04b56876f663cc270765e(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:43,178 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:43,178 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., storeName=810791807ce3c9f7e9f14187121bf792/B, priority=12, startTime=1682042143146; duration=0sec 2023-04-21 01:55:43,178 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:43,178 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:55:43,179 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:55:43,179 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 810791807ce3c9f7e9f14187121bf792/C is initiating minor compaction (all files) 2023-04-21 01:55:43,179 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 810791807ce3c9f7e9f14187121bf792/C in TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:43,179 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/26b3a5f2ad164c95a3404db4aa0155c9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/379abf706ff4411eb29e6ee0148d8cd6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/d0e078b59a0a4c2d9c7fa2eea2c3b42b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/94639410d832481cad9507b3fde09f92] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp, totalSize=48.3 K 2023-04-21 01:55:43,179 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 26b3a5f2ad164c95a3404db4aa0155c9, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1682042138041 2023-04-21 01:55:43,179 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 379abf706ff4411eb29e6ee0148d8cd6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=462, earliestPutTs=1682042138261 2023-04-21 01:55:43,180 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting d0e078b59a0a4c2d9c7fa2eea2c3b42b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1682042140470 2023-04-21 01:55:43,180 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 94639410d832481cad9507b3fde09f92, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1682042141165 2023-04-21 01:55:43,193 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 810791807ce3c9f7e9f14187121bf792#C#compaction#410 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:43,193 DEBUG [Thread-2525] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a5ab074 to 127.0.0.1:58383 2023-04-21 01:55:43,193 DEBUG [Thread-2525] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:55:43,193 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-21 01:55:43,193 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 17 2023-04-21 01:55:43,194 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 17 2023-04-21 01:55:43,194 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 15 2023-04-21 01:55:43,194 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 357 2023-04-21 01:55:43,194 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 16 2023-04-21 01:55:43,194 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-21 01:55:43,194 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-21 01:55:43,194 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(402): scanned 1800 2023-04-21 01:55:43,194 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(403): verified 5400 rows 2023-04-21 01:55:43,194 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(402): scanned 1800 2023-04-21 01:55:43,194 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(403): verified 5400 rows 2023-04-21 01:55:43,194 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(402): scanned 1787 2023-04-21 01:55:43,194 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(403): verified 5361 rows 2023-04-21 01:55:43,194 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(402): scanned 1804 2023-04-21 01:55:43,194 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(403): verified 5411 rows 2023-04-21 01:55:43,194 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(402): scanned 1789 2023-04-21 01:55:43,194 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(403): verified 5367 rows 2023-04-21 01:55:43,194 INFO [Listener at localhost/39939] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-21 01:55:43,194 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0fd3d54e to 127.0.0.1:58383 2023-04-21 01:55:43,194 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:55:43,197 INFO [Listener at localhost/39939] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-21 01:55:43,197 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-21 01:55:43,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-21 01:55:43,203 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042143202"}]},"ts":"1682042143202"} 2023-04-21 01:55:43,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-21 01:55:43,204 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-21 01:55:43,205 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-21 01:55:43,206 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=810791807ce3c9f7e9f14187121bf792, UNASSIGN}] 2023-04-21 01:55:43,209 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=810791807ce3c9f7e9f14187121bf792, UNASSIGN 2023-04-21 01:55:43,209 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=810791807ce3c9f7e9f14187121bf792, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:43,210 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042143209"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682042143209"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682042143209"}]},"ts":"1682042143209"} 2023-04-21 01:55:43,210 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399}] 2023-04-21 01:55:43,218 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/f56443d8a318496680408c3b04e6e1e3 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/f56443d8a318496680408c3b04e6e1e3 2023-04-21 01:55:43,221 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 810791807ce3c9f7e9f14187121bf792/A of 810791807ce3c9f7e9f14187121bf792 into f56443d8a318496680408c3b04e6e1e3(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:43,221 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:43,221 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., storeName=810791807ce3c9f7e9f14187121bf792/A, priority=12, startTime=1682042143146; duration=0sec 2023-04-21 01:55:43,221 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:43,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-21 01:55:43,362 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:43,362 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 810791807ce3c9f7e9f14187121bf792, disabling compactions & flushes 2023-04-21 01:55:43,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 1 compactions to complete for region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:43,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-21 01:55:43,611 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/2dad46262b754b2c93febfd430b584a5 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/2dad46262b754b2c93febfd430b584a5 2023-04-21 01:55:43,615 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 810791807ce3c9f7e9f14187121bf792/C of 810791807ce3c9f7e9f14187121bf792 into 2dad46262b754b2c93febfd430b584a5(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:43,616 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:43,616 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:43,616 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792., storeName=810791807ce3c9f7e9f14187121bf792/C, priority=12, startTime=1682042143146; duration=0sec 2023-04-21 01:55:43,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:43,616 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:43,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. after waiting 0 ms 2023-04-21 01:55:43,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:43,616 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 810791807ce3c9f7e9f14187121bf792 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-21 01:55:43,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=A 2023-04-21 01:55:43,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:43,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=B 2023-04-21 01:55:43,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:43,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 810791807ce3c9f7e9f14187121bf792, store=C 2023-04-21 01:55:43,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:43,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-21 01:55:44,029 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421c8f91a20ac764475b9802e95e0934cd8 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421c8f91a20ac764475b9802e95e0934cd8 2023-04-21 01:55:44,034 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=507, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/6a102049564a4c46a30b96c6b5df4b5d 2023-04-21 01:55:44,042 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/fd1176c9df904a9193e2036cb069a0a5 2023-04-21 01:55:44,050 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/93756b9b9be54470af4f13baeedabf28 2023-04-21 01:55:44,053 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/A/6a102049564a4c46a30b96c6b5df4b5d as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/6a102049564a4c46a30b96c6b5df4b5d 2023-04-21 01:55:44,056 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/6a102049564a4c46a30b96c6b5df4b5d, entries=100, sequenceid=507, filesize=18.6 K 2023-04-21 01:55:44,057 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/B/fd1176c9df904a9193e2036cb069a0a5 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/fd1176c9df904a9193e2036cb069a0a5 2023-04-21 01:55:44,060 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/fd1176c9df904a9193e2036cb069a0a5, entries=100, sequenceid=507, filesize=9.5 K 2023-04-21 01:55:44,060 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/.tmp/C/93756b9b9be54470af4f13baeedabf28 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/93756b9b9be54470af4f13baeedabf28 2023-04-21 01:55:44,063 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/93756b9b9be54470af4f13baeedabf28, entries=100, sequenceid=507, filesize=9.5 K 2023-04-21 01:55:44,064 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 810791807ce3c9f7e9f14187121bf792 in 448ms, sequenceid=507, compaction requested=false 2023-04-21 01:55:44,064 DEBUG [StoreCloser-TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/0c4e668a462e4adaa799416a2e05e8c3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/427a9d8f015741ee8bcd7f021f9b62ae, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/3942af1dbd594db58c8b4cd4696b33b3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/9176f1825be54348862e2e6f4edbd0e7] to archive 2023-04-21 01:55:44,065 DEBUG [StoreCloser-TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-21 01:55:44,066 DEBUG [StoreCloser-TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/0c4e668a462e4adaa799416a2e05e8c3 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/0c4e668a462e4adaa799416a2e05e8c3 2023-04-21 01:55:44,067 DEBUG [StoreCloser-TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/427a9d8f015741ee8bcd7f021f9b62ae to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/427a9d8f015741ee8bcd7f021f9b62ae 2023-04-21 01:55:44,068 DEBUG [StoreCloser-TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/3942af1dbd594db58c8b4cd4696b33b3 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/3942af1dbd594db58c8b4cd4696b33b3 2023-04-21 01:55:44,069 DEBUG [StoreCloser-TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/9176f1825be54348862e2e6f4edbd0e7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/9176f1825be54348862e2e6f4edbd0e7 2023-04-21 01:55:44,070 DEBUG [StoreCloser-TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/07e8b394c2c74bacb060ce28586f77ae, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/27d2a0f2fee642ddb2c683ee48eab658, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/b45516b9e6f34bbbbbd9484ab90d9794, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/eac52abfadc7445f9d31f2a1b170be37] to archive 2023-04-21 01:55:44,071 DEBUG [StoreCloser-TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-21 01:55:44,072 DEBUG [StoreCloser-TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/07e8b394c2c74bacb060ce28586f77ae to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/07e8b394c2c74bacb060ce28586f77ae 2023-04-21 01:55:44,073 DEBUG [StoreCloser-TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/27d2a0f2fee642ddb2c683ee48eab658 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/27d2a0f2fee642ddb2c683ee48eab658 2023-04-21 01:55:44,074 DEBUG [StoreCloser-TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/b45516b9e6f34bbbbbd9484ab90d9794 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/b45516b9e6f34bbbbbd9484ab90d9794 2023-04-21 01:55:44,074 DEBUG [StoreCloser-TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/eac52abfadc7445f9d31f2a1b170be37 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/eac52abfadc7445f9d31f2a1b170be37 2023-04-21 01:55:44,075 DEBUG [StoreCloser-TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/26b3a5f2ad164c95a3404db4aa0155c9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/379abf706ff4411eb29e6ee0148d8cd6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/d0e078b59a0a4c2d9c7fa2eea2c3b42b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/94639410d832481cad9507b3fde09f92] to archive 2023-04-21 01:55:44,076 DEBUG [StoreCloser-TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-21 01:55:44,077 DEBUG [StoreCloser-TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/26b3a5f2ad164c95a3404db4aa0155c9 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/26b3a5f2ad164c95a3404db4aa0155c9 2023-04-21 01:55:44,077 DEBUG [StoreCloser-TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/379abf706ff4411eb29e6ee0148d8cd6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/379abf706ff4411eb29e6ee0148d8cd6 2023-04-21 01:55:44,078 DEBUG [StoreCloser-TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/d0e078b59a0a4c2d9c7fa2eea2c3b42b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/d0e078b59a0a4c2d9c7fa2eea2c3b42b 2023-04-21 01:55:44,079 DEBUG [StoreCloser-TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/94639410d832481cad9507b3fde09f92 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/94639410d832481cad9507b3fde09f92 2023-04-21 01:55:44,082 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/recovered.edits/510.seqid, newMaxSeqId=510, maxSeqId=4 2023-04-21 01:55:44,084 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792. 2023-04-21 01:55:44,084 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 810791807ce3c9f7e9f14187121bf792: 2023-04-21 01:55:44,085 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:44,085 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=810791807ce3c9f7e9f14187121bf792, regionState=CLOSED 2023-04-21 01:55:44,086 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042144085"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682042144085"}]},"ts":"1682042144085"} 2023-04-21 01:55:44,087 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-21 01:55:44,087 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 810791807ce3c9f7e9f14187121bf792, server=jenkins-hbase4.apache.org,40131,1682042019399 in 876 msec 2023-04-21 01:55:44,088 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-21 01:55:44,089 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=810791807ce3c9f7e9f14187121bf792, UNASSIGN in 881 msec 2023-04-21 01:55:44,089 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042144089"}]},"ts":"1682042144089"} 2023-04-21 01:55:44,090 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-21 01:55:44,091 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-21 01:55:44,092 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 895 msec 2023-04-21 01:55:44,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-21 01:55:44,307 INFO [Listener at localhost/39939] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-21 01:55:44,308 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-21 01:55:44,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:55:44,310 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:55:44,311 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:55:44,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-21 01:55:44,315 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:44,317 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A, FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B, FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C, FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/recovered.edits] 2023-04-21 01:55:44,319 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/6a102049564a4c46a30b96c6b5df4b5d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/6a102049564a4c46a30b96c6b5df4b5d 2023-04-21 01:55:44,320 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/f56443d8a318496680408c3b04e6e1e3 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/A/f56443d8a318496680408c3b04e6e1e3 2023-04-21 01:55:44,323 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/4fc5447597c04b56876f663cc270765e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/4fc5447597c04b56876f663cc270765e 2023-04-21 01:55:44,324 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/fd1176c9df904a9193e2036cb069a0a5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/B/fd1176c9df904a9193e2036cb069a0a5 2023-04-21 01:55:44,326 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/2dad46262b754b2c93febfd430b584a5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/2dad46262b754b2c93febfd430b584a5 2023-04-21 01:55:44,327 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/93756b9b9be54470af4f13baeedabf28 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/C/93756b9b9be54470af4f13baeedabf28 2023-04-21 01:55:44,329 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/recovered.edits/510.seqid to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792/recovered.edits/510.seqid 2023-04-21 01:55:44,329 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/810791807ce3c9f7e9f14187121bf792 2023-04-21 01:55:44,329 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-21 01:55:44,330 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-21 01:55:44,330 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-21 01:55:44,333 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304211539a178fce845e9979418651fd41de6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304211539a178fce845e9979418651fd41de6 2023-04-21 01:55:44,334 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304211651fca4118343eabf1014a0081216d7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304211651fca4118343eabf1014a0081216d7 2023-04-21 01:55:44,335 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042116f5c6b613364f019cc9b0808471048e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042116f5c6b613364f019cc9b0808471048e 2023-04-21 01:55:44,336 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304212ef1fb7229974ad98df8d2653e100d80 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304212ef1fb7229974ad98df8d2653e100d80 2023-04-21 01:55:44,337 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042130cb26c1241044a4a69a0134296fb224 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042130cb26c1241044a4a69a0134296fb224 2023-04-21 01:55:44,338 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421486ce04eae2f49a8a8525a691aeb9a4e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421486ce04eae2f49a8a8525a691aeb9a4e 2023-04-21 01:55:44,338 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042148b20b0b16ee4e65937eee1ebc358dea to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042148b20b0b16ee4e65937eee1ebc358dea 2023-04-21 01:55:44,339 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042149c05de38762408cb7c131d9afe5f5e5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042149c05de38762408cb7c131d9afe5f5e5 2023-04-21 01:55:44,340 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304215a92ac26f0c241418354f0c3ca58147c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304215a92ac26f0c241418354f0c3ca58147c 2023-04-21 01:55:44,341 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304215c1c5658e03e46a796e151ebad64e9ba to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304215c1c5658e03e46a796e151ebad64e9ba 2023-04-21 01:55:44,342 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304215c31b05b152948e1b49a70bffdd78e89 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304215c31b05b152948e1b49a70bffdd78e89 2023-04-21 01:55:44,343 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304216704a3a89c9a4ecbb9242e8da82772a4 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304216704a3a89c9a4ecbb9242e8da82772a4 2023-04-21 01:55:44,344 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304216ad32ac92d8040a88a47572b19aab073 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304216ad32ac92d8040a88a47572b19aab073 2023-04-21 01:55:44,345 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304219715a252c32a4ed394108f7707fcea66 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304219715a252c32a4ed394108f7707fcea66 2023-04-21 01:55:44,345 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304219bd41941d2d74be18e3a75c3da2567e5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304219bd41941d2d74be18e3a75c3da2567e5 2023-04-21 01:55:44,346 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421c8f91a20ac764475b9802e95e0934cd8 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421c8f91a20ac764475b9802e95e0934cd8 2023-04-21 01:55:44,347 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421cc25ec09ffa4498c9c6d7c5a3fcbd469 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421cc25ec09ffa4498c9c6d7c5a3fcbd469 2023-04-21 01:55:44,348 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421cfae0cfa8eeb43d4b95117565c53e097 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421cfae0cfa8eeb43d4b95117565c53e097 2023-04-21 01:55:44,349 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421f1f3ab91011642cbb33439e8dd800ff5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421f1f3ab91011642cbb33439e8dd800ff5 2023-04-21 01:55:44,350 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421fd1df3e711574ab1845bb90842f1cc8d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421fd1df3e711574ab1845bb90842f1cc8d 2023-04-21 01:55:44,350 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-21 01:55:44,352 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:55:44,354 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-21 01:55:44,355 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-21 01:55:44,356 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:55:44,356 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-21 01:55:44,356 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682042144356"}]},"ts":"9223372036854775807"} 2023-04-21 01:55:44,358 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-21 01:55:44,358 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 810791807ce3c9f7e9f14187121bf792, NAME => 'TestAcidGuarantees,,1682042118297.810791807ce3c9f7e9f14187121bf792.', STARTKEY => '', ENDKEY => ''}] 2023-04-21 01:55:44,358 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-21 01:55:44,358 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682042144358"}]},"ts":"9223372036854775807"} 2023-04-21 01:55:44,359 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-21 01:55:44,362 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:55:44,363 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 54 msec 2023-04-21 01:55:44,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-21 01:55:44,412 INFO [Listener at localhost/39939] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-21 01:55:44,426 INFO [Listener at localhost/39939] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=240 (was 226) - Thread LEAK? -, OpenFileDescriptor=552 (was 569), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=440 (was 401) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=8966 (was 8651) - AvailableMemoryMB LEAK? - 2023-04-21 01:55:44,440 INFO [Listener at localhost/39939] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=240, OpenFileDescriptor=552, MaxFileDescriptor=60000, SystemLoadAverage=440, ProcessCount=169, AvailableMemoryMB=8966 2023-04-21 01:55:44,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-21 01:55:44,443 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-21 01:55:44,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-21 01:55:44,446 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-21 01:55:44,446 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-21 01:55:44,446 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-21 01:55:44,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-21 01:55:44,448 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:44,448 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82 empty. 2023-04-21 01:55:44,448 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:44,448 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-21 01:55:44,455 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-21 01:55:44,461 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 82ecbe07d2c67bb21d6485daedbb8c82, NAME => 'TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp 2023-04-21 01:55:44,479 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-21 01:55:44,480 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 82ecbe07d2c67bb21d6485daedbb8c82, disabling compactions & flushes 2023-04-21 01:55:44,480 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:44,480 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:44,480 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. after waiting 0 ms 2023-04-21 01:55:44,480 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:44,480 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:44,480 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:44,482 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-21 01:55:44,483 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042144482"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682042144482"}]},"ts":"1682042144482"} 2023-04-21 01:55:44,484 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-21 01:55:44,484 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-21 01:55:44,484 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042144484"}]},"ts":"1682042144484"} 2023-04-21 01:55:44,485 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-21 01:55:44,488 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=82ecbe07d2c67bb21d6485daedbb8c82, ASSIGN}] 2023-04-21 01:55:44,489 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=82ecbe07d2c67bb21d6485daedbb8c82, ASSIGN 2023-04-21 01:55:44,490 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=82ecbe07d2c67bb21d6485daedbb8c82, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40131,1682042019399; forceNewPlan=false, retain=false 2023-04-21 01:55:44,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-21 01:55:44,641 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=82ecbe07d2c67bb21d6485daedbb8c82, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:44,642 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042144641"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682042144641"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682042144641"}]},"ts":"1682042144641"} 2023-04-21 01:55:44,643 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399}] 2023-04-21 01:55:44,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-21 01:55:44,798 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:44,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 82ecbe07d2c67bb21d6485daedbb8c82, NAME => 'TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.', STARTKEY => '', ENDKEY => ''} 2023-04-21 01:55:44,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:44,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-21 01:55:44,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:44,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:44,800 INFO [StoreOpener-82ecbe07d2c67bb21d6485daedbb8c82-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:44,801 DEBUG [StoreOpener-82ecbe07d2c67bb21d6485daedbb8c82-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A 2023-04-21 01:55:44,801 DEBUG [StoreOpener-82ecbe07d2c67bb21d6485daedbb8c82-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A 2023-04-21 01:55:44,801 INFO [StoreOpener-82ecbe07d2c67bb21d6485daedbb8c82-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:55:44,801 INFO [StoreOpener-82ecbe07d2c67bb21d6485daedbb8c82-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 82ecbe07d2c67bb21d6485daedbb8c82 columnFamilyName A 2023-04-21 01:55:44,802 INFO [StoreOpener-82ecbe07d2c67bb21d6485daedbb8c82-1] regionserver.HStore(310): Store=82ecbe07d2c67bb21d6485daedbb8c82/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:55:44,802 INFO [StoreOpener-82ecbe07d2c67bb21d6485daedbb8c82-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:44,803 DEBUG [StoreOpener-82ecbe07d2c67bb21d6485daedbb8c82-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B 2023-04-21 01:55:44,803 DEBUG [StoreOpener-82ecbe07d2c67bb21d6485daedbb8c82-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B 2023-04-21 01:55:44,803 INFO [StoreOpener-82ecbe07d2c67bb21d6485daedbb8c82-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:55:44,803 INFO [StoreOpener-82ecbe07d2c67bb21d6485daedbb8c82-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 82ecbe07d2c67bb21d6485daedbb8c82 columnFamilyName B 2023-04-21 01:55:44,804 INFO [StoreOpener-82ecbe07d2c67bb21d6485daedbb8c82-1] regionserver.HStore(310): Store=82ecbe07d2c67bb21d6485daedbb8c82/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:55:44,804 INFO [StoreOpener-82ecbe07d2c67bb21d6485daedbb8c82-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:44,805 DEBUG [StoreOpener-82ecbe07d2c67bb21d6485daedbb8c82-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C 2023-04-21 01:55:44,805 DEBUG [StoreOpener-82ecbe07d2c67bb21d6485daedbb8c82-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C 2023-04-21 01:55:44,805 INFO [StoreOpener-82ecbe07d2c67bb21d6485daedbb8c82-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:55:44,805 INFO [StoreOpener-82ecbe07d2c67bb21d6485daedbb8c82-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 82ecbe07d2c67bb21d6485daedbb8c82 columnFamilyName C 2023-04-21 01:55:44,805 INFO [StoreOpener-82ecbe07d2c67bb21d6485daedbb8c82-1] regionserver.HStore(310): Store=82ecbe07d2c67bb21d6485daedbb8c82/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:55:44,806 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:44,806 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:44,806 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:44,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-21 01:55:44,809 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:44,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-21 01:55:44,810 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 82ecbe07d2c67bb21d6485daedbb8c82; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66109667, jitterRate=-0.014889195561408997}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-21 01:55:44,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:44,811 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., pid=49, masterSystemTime=1682042144795 2023-04-21 01:55:44,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:44,812 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:44,813 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=82ecbe07d2c67bb21d6485daedbb8c82, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:44,813 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042144812"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682042144812"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682042144812"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682042144812"}]},"ts":"1682042144812"} 2023-04-21 01:55:44,814 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-21 01:55:44,815 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 in 171 msec 2023-04-21 01:55:44,816 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-21 01:55:44,816 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=82ecbe07d2c67bb21d6485daedbb8c82, ASSIGN in 327 msec 2023-04-21 01:55:44,816 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-21 01:55:44,816 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042144816"}]},"ts":"1682042144816"} 2023-04-21 01:55:44,817 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-21 01:55:44,819 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-21 01:55:44,820 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 376 msec 2023-04-21 01:55:45,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-21 01:55:45,050 INFO [Listener at localhost/39939] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-21 01:55:45,051 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x1c29757d to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:55:45,057 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@420f945a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:55:45,058 DEBUG [Listener at localhost/39939] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:55:45,061 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60362, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:55:45,062 DEBUG [Listener at localhost/39939] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-21 01:55:45,064 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58700, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-21 01:55:45,065 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x06edd017 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:55:45,069 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51b8df2a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:55:45,069 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x0fb168ce to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:55:45,074 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67346c47, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:55:45,075 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x461f5f3f to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:55:45,079 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@426cb82a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:55:45,079 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x63093e21 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:55:45,086 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f5363d8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:55:45,086 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x0b475d8d to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:55:45,090 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@211582f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:55:45,091 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x65816b6e to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:55:45,094 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@680e773e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:55:45,094 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x653bb8fc to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:55:45,100 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b8f652e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:55:45,100 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x720d29f5 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:55:45,106 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ac98310, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:55:45,106 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x3011deb3 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:55:45,111 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2b480bc5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:55:45,111 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x6e8824d7 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:55:45,115 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50529bf3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:55:45,121 DEBUG [hconnection-0x70dab86f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:55:45,123 DEBUG [hconnection-0x44805521-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:55:45,125 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60380, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:55:45,130 DEBUG [hconnection-0x5cdb76cf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:55:45,130 DEBUG [hconnection-0x7073f0fc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:55:45,130 DEBUG [hconnection-0x75d4e3dc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:55:45,132 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60376, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:55:45,132 DEBUG [hconnection-0x45896b70-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:55:45,133 DEBUG [hconnection-0x51b6c389-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:55:45,138 DEBUG [hconnection-0x29158438-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:55:45,138 DEBUG [hconnection-0x5f839980-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:55:45,139 DEBUG [hconnection-0x5bab7893-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:55:45,139 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60382, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:55:45,140 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60410, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:55:45,143 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60420, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:55:45,144 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60398, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:55:45,144 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60422, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:55:45,148 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60438, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:55:45,150 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:45,150 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60448, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:55:45,159 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60462, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:55:45,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:45,168 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-21 01:55:45,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:45,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:45,169 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:45,169 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:45,169 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:45,169 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:45,169 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:45,169 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,169 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:45,169 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,169 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:45,169 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:45,169 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:45,170 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:45,172 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,172 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:45,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,173 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,173 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:45,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:45,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:45,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:45,176 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:45,176 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:45,176 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:45,177 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:45,177 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:45,177 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:45,177 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:45,177 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:45,177 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:45,177 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:45,177 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:45,177 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,177 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:45,177 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:45,185 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:45,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:45,186 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:45,186 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,186 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:45,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:45,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:45,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:45,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e96358f[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:45,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:45,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,193 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042205190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,194 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682042145189, max timestamp=1682042145191Num uniques 859; 2023-04-21 01:55:45,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682042145189, max timestamp=1682042145191Num uniques 859; 2023-04-21 01:55:45,195 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:45,196 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:45,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:55:45,196 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,196 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:45,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682042145189, max timestamp=1682042145191Num uniques 859; 2023-04-21 01:55:45,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:55:45,196 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:45,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:55:45,198 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:45,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:45,198 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:45,198 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:45,198 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:45,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:45,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:45,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:45,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:45,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:45,200 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:45,200 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:45,200 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:45,201 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:45,201 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:45,202 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042205202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,203 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,203 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:45,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042205202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042205201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,203 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042205203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:45,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:45,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,203 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:45,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:45,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:45,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:45,205 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:45,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,211 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:45,211 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:45,211 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:45,211 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,211 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,211 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,211 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,211 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,212 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:45,211 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:45,211 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/0c7e3c7287fd45f58d3510b3713ff792 2023-04-21 01:55:45,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:45,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:45,212 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:45,212 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:45,212 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,212 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:45,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:45,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:45,241 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/885a1c3c1cf34c01bd6bad99d90ce34a 2023-04-21 01:55:45,265 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/4acfb3cc437f4afb8dddd95c05908554 2023-04-21 01:55:45,271 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/0c7e3c7287fd45f58d3510b3713ff792 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/0c7e3c7287fd45f58d3510b3713ff792 2023-04-21 01:55:45,275 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/0c7e3c7287fd45f58d3510b3713ff792, entries=150, sequenceid=13, filesize=11.6 K 2023-04-21 01:55:45,275 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/885a1c3c1cf34c01bd6bad99d90ce34a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/885a1c3c1cf34c01bd6bad99d90ce34a 2023-04-21 01:55:45,279 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/885a1c3c1cf34c01bd6bad99d90ce34a, entries=150, sequenceid=13, filesize=11.6 K 2023-04-21 01:55:45,279 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/4acfb3cc437f4afb8dddd95c05908554 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/4acfb3cc437f4afb8dddd95c05908554 2023-04-21 01:55:45,282 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/4acfb3cc437f4afb8dddd95c05908554, entries=150, sequenceid=13, filesize=11.6 K 2023-04-21 01:55:45,283 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for 82ecbe07d2c67bb21d6485daedbb8c82 in 115ms, sequenceid=13, compaction requested=false 2023-04-21 01:55:45,283 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:45,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:45,302 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-21 01:55:45,303 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:45,303 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:45,303 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:45,303 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:45,303 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:45,303 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:45,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,313 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:45,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042205307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:45,315 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:45,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042205308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042205309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042205311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042205313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:45,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:45,340 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:45,341 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:45,341 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:45,341 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:45,341 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,341 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,342 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:45,342 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:45,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:45,343 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,343 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:45,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:45,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:45,344 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/25c0a161746540f0bc6474b64ef5e060 2023-04-21 01:55:45,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:45,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:45,352 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:45,352 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:45,352 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:45,356 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:45,356 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:45,356 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:45,356 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:45,356 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:45,356 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:45,357 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:45,357 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:45,357 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,357 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:45,357 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:45,358 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:45,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:45,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:45,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:45,359 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:45,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:45,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@377b5fc9[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:45,360 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:45,360 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,363 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,363 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,363 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:45,363 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,363 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:45,363 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:45,363 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:45,363 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:45,365 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,365 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:45,365 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:45,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:45,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:45,365 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:45,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:45,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:45,368 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:45,368 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:45,368 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:45,368 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:45,368 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:45,369 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,369 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,369 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:45,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:45,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:45,370 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,371 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,375 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,375 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,376 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,376 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:45,376 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:45,376 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:45,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:45,376 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,376 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:45,376 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:45,376 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:45,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:45,376 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,376 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:45,376 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:45,376 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,376 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042205414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042205420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042205420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042205420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042205420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,476 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:45,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:45,478 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:45,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:45,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:45,479 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:45,479 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:45,479 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:45,479 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:45,480 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,480 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,482 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,482 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:45,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:45,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:45,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:45,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:45,483 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,483 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:45,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:45,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:45,488 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:45,488 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:45,488 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:45,489 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:45,489 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:45,489 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:45,489 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:45,489 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:45,489 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:45,489 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:45,489 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:45,489 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,489 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:45,489 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:45,490 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,490 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:45,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:45,491 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,491 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:45,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:45,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:45,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:45,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:45,492 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:45,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52da5126[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:45,492 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,493 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,493 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:45,493 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:45,493 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:45,493 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,493 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:45,494 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,494 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:45,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:45,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:45,494 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:45,494 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:45,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:45,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:45,497 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:45,497 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:45,497 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:45,497 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:45,497 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:45,499 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,499 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:45,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,499 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:45,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:45,499 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,500 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:45,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,502 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,502 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,502 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,502 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,502 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:45,502 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:45,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:45,502 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,503 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:45,503 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:45,503 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:45,502 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:45,503 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:45,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:45,503 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:45,503 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:45,603 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:45,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:45,605 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:45,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:45,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:45,606 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:45,606 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:45,606 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:45,606 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:45,607 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,607 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,608 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:45,608 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:45,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:45,608 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,608 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:45,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:45,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:45,610 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:45,610 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:45,610 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:45,610 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:45,610 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:45,610 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:45,610 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:45,610 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:45,610 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:45,610 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:45,610 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:45,610 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,610 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:45,610 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:45,612 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,612 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:45,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:45,612 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,612 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:45,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:45,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:45,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:45,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:45,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78ec8ea6[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:45,614 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:45,614 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,616 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,616 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:45,616 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:45,616 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:45,616 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,616 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:45,617 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,617 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:45,617 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:45,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:45,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:45,617 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:45,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:45,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:45,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:45,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:45,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:45,622 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:45,622 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:45,622 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:45,622 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:45,622 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:45,623 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,623 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,624 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:45,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:45,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:45,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:45,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:45,624 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:45,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,625 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:45,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042205626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042205629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042205629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042205629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042205630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,632 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,632 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,632 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:45,632 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:45,632 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:45,632 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:45,632 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,632 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,632 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,632 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,632 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:45,632 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:45,632 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:45,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:45,632 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:45,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:45,733 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:45,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:45,735 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:45,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:45,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:45,736 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:45,736 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:45,736 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:45,736 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:45,736 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,736 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,737 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,737 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:45,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:45,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:45,738 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,738 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:45,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:45,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:45,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:45,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:45,739 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:45,739 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:45,739 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:45,739 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:45,739 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:45,739 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:45,739 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:45,739 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:45,739 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:45,739 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:45,739 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:45,739 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:45,741 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,741 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:45,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:45,741 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,741 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:45,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:45,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:45,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:45,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:45,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47e9e679[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:45,743 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:45,743 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,745 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,746 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:45,746 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:45,746 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:45,746 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,746 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:45,747 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:45,747 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:45,747 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,747 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:45,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:45,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:45,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:45,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:45,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:45,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:45,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:45,749 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:45,749 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:45,749 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:45,749 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:45,750 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:45,751 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,751 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:45,751 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:45,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:45,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:45,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:45,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:45,752 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:45,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,753 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:45,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,755 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,755 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,756 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,756 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,756 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:45,756 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:45,756 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:45,756 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:45,756 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:45,756 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:45,756 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:45,756 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:45,756 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,756 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:45,785 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/55f5a950fec74790be7d8a2da0f0f1f1 2023-04-21 01:55:45,856 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:45,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:45,858 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:45,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:45,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:45,860 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:45,860 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:45,860 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:45,860 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:45,860 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,860 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,862 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:45,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:45,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:45,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,863 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,863 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,863 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:45,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:45,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,863 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:45,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:45,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:45,864 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:45,864 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:45,864 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:45,864 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:45,864 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:45,864 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:45,864 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:45,864 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:45,864 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:45,864 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:45,864 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:45,864 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,864 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:45,865 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:45,866 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,866 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:45,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:45,866 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,866 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:45,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:45,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:45,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:45,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a016c5a[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:45,873 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:45,873 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,874 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,874 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:45,874 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:45,874 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:45,874 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,874 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:45,877 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:45,877 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:45,877 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,877 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:45,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:45,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:45,880 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:45,880 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:45,880 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:45,880 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:45,880 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:45,884 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,884 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,884 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:45,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:45,885 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,886 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,888 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,888 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,888 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,888 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,888 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:45,888 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:45,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:45,888 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:45,888 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:45,889 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:45,888 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,889 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:45,889 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,889 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:45,889 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:45,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:45,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042205932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042205933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042205933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042205933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:45,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042205934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,989 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:45,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:45,991 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:45,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:45,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:45,993 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:45,993 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:45,993 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:45,993 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:45,993 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,993 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,994 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:45,994 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:45,995 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:45,995 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:45,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:45,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:45,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:46,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:46,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:46,002 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:46,002 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:46,002 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:46,004 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:46,004 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:46,004 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:46,004 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:46,004 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:46,004 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:46,004 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:46,004 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:46,004 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,004 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:46,004 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:46,006 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:46,006 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:46,007 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,007 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:46,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:46,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:46,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:46,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a524b5d[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:46,008 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:46,008 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,010 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,010 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:46,010 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:46,010 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:46,012 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,012 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:46,014 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:46,014 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:46,014 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,014 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:46,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:46,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:46,018 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:46,018 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:46,018 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:46,018 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:46,018 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:46,020 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,020 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:46,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:46,020 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:46,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,021 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:46,024 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,027 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:46,027 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:46,027 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:46,027 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:46,027 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:46,027 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:46,028 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,028 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,028 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,028 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:46,028 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:46,028 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:46,028 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:46,028 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:46,129 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:46,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:46,131 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:46,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:46,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:46,132 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:46,132 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:46,132 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:46,132 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:46,133 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,133 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:46,135 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,135 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:46,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:46,135 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,135 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:46,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:46,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:46,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:46,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:46,136 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:46,136 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:46,136 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:46,137 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:46,137 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:46,137 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:46,137 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:46,137 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:46,137 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:46,137 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:46,137 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:46,137 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,137 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:46,137 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:46,141 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,141 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:46,141 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,142 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:46,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:46,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3657790f[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:46,144 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:46,144 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,148 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,148 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:46,148 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:46,148 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:46,148 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,148 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:46,151 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:46,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:46,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:46,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:46,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:46,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:46,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:46,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:46,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:46,154 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:46,154 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:46,154 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:46,154 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:46,154 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:46,156 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,156 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,156 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:46,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:46,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:46,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:46,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,157 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:46,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,158 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:46,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,161 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:46,161 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:46,161 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:46,161 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:46,161 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:46,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:46,161 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,162 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,162 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,162 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:46,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:46,162 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:46,163 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:46,163 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:46,163 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:46,163 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:46,195 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/b3ddb399295740d89551fc99e1cd618a 2023-04-21 01:55:46,199 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/25c0a161746540f0bc6474b64ef5e060 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/25c0a161746540f0bc6474b64ef5e060 2023-04-21 01:55:46,203 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/25c0a161746540f0bc6474b64ef5e060, entries=200, sequenceid=49, filesize=13.9 K 2023-04-21 01:55:46,203 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/55f5a950fec74790be7d8a2da0f0f1f1 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/55f5a950fec74790be7d8a2da0f0f1f1 2023-04-21 01:55:46,206 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/55f5a950fec74790be7d8a2da0f0f1f1, entries=150, sequenceid=49, filesize=11.6 K 2023-04-21 01:55:46,207 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/b3ddb399295740d89551fc99e1cd618a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/b3ddb399295740d89551fc99e1cd618a 2023-04-21 01:55:46,210 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/b3ddb399295740d89551fc99e1cd618a, entries=150, sequenceid=49, filesize=11.6 K 2023-04-21 01:55:46,211 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 82ecbe07d2c67bb21d6485daedbb8c82 in 909ms, sequenceid=49, compaction requested=false 2023-04-21 01:55:46,211 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:46,263 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:46,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:46,265 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:46,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:46,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:46,266 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:46,266 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:46,266 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:46,266 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:46,267 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,267 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:46,268 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:46,268 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:46,268 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,268 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:46,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:46,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:46,269 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:46,269 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:46,269 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:46,270 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:46,270 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:46,270 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:46,270 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:46,270 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:46,270 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:46,270 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-21 01:55:46,270 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:46,270 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:46,271 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:46,271 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:46,271 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:46,271 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:46,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:46,442 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:46,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682042146449, max timestamp=1682042146460Num uniques 859; 2023-04-21 01:55:46,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-21 01:55:46,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682042146449, max timestamp=1682042146460Num uniques 859; 2023-04-21 01:55:46,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-21 01:55:46,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682042146449, max timestamp=1682042146460Num uniques 859; 2023-04-21 01:55:46,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-21 01:55:46,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:46,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042206478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:46,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042206479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:46,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042206480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:46,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042206484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:46,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042206485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,590 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:46,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042206586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:46,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042206586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:46,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042206591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:46,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042206592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:46,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042206592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,679 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/fbf94c7344cb49e583bd4436e2030160 2023-04-21 01:55:46,688 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/e729994638cd45a495e2b0c7774a8ad0 2023-04-21 01:55:46,697 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/1e7ce63ba7ba4bda83b09afe66644e48 2023-04-21 01:55:46,705 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/fbf94c7344cb49e583bd4436e2030160 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/fbf94c7344cb49e583bd4436e2030160 2023-04-21 01:55:46,710 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/fbf94c7344cb49e583bd4436e2030160, entries=100, sequenceid=59, filesize=9.3 K 2023-04-21 01:55:46,711 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/e729994638cd45a495e2b0c7774a8ad0 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/e729994638cd45a495e2b0c7774a8ad0 2023-04-21 01:55:46,715 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/e729994638cd45a495e2b0c7774a8ad0, entries=100, sequenceid=59, filesize=9.3 K 2023-04-21 01:55:46,715 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/1e7ce63ba7ba4bda83b09afe66644e48 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1e7ce63ba7ba4bda83b09afe66644e48 2023-04-21 01:55:46,719 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1e7ce63ba7ba4bda83b09afe66644e48, entries=100, sequenceid=59, filesize=9.3 K 2023-04-21 01:55:46,720 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 82ecbe07d2c67bb21d6485daedbb8c82 in 450ms, sequenceid=59, compaction requested=true 2023-04-21 01:55:46,720 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:46,720 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:46,720 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:46,720 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:46,720 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,720 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:46,721 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:46,722 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,722 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:46,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:46,723 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,723 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:46,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:46,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:46,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:46,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:46,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e058b92[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:46,724 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:46,724 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,727 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,727 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:46,727 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:46,727 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:46,730 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:46,730 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:46,730 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,730 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:46,730 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:46,730 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:46,730 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:46,730 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:46,730 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:46,730 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:46,730 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:46,732 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:46,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:46,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:46,733 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,733 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,733 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:46,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:46,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:46,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:46,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,736 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:46,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,738 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,738 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:46,738 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:46,738 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:46,738 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:46,738 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,738 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,738 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,738 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:46,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:46,738 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:46,738 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:46,738 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:46,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:46,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:46,739 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:46,739 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:46,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:46,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:46,796 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-21 01:55:46,797 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:46,797 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:46,797 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:46,797 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:46,797 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:46,797 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:46,805 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/09b09166d5ca41afaa29427354ac5934 2023-04-21 01:55:46,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:46,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:46,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042206812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042206813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:46,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042206813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:46,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042206814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:46,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042206814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,839 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:46,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:46,841 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:46,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:46,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:46,842 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:46,842 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:46,842 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:46,842 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:46,842 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,842 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:46,843 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:46,843 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:46,844 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,844 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:46,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:46,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:46,845 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:46,845 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:46,845 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:46,845 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:46,845 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:46,845 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:46,845 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:46,845 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:46,845 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:46,845 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:46,845 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:46,845 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:46,845 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:46,846 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,846 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:46,846 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:46,848 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:46,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:46,849 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:46,850 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,850 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:46,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:46,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:46,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:46,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fc743b4[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:46,851 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:46,852 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,853 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,853 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,853 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:46,854 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:46,854 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:46,854 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:46,855 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:46,855 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,855 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:46,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:46,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:46,855 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:46,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:46,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:46,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:46,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:46,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:46,857 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:46,857 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:46,857 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:46,857 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:46,858 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:46,859 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,859 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:46,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:46,859 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:46,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:46,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:46,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:46,860 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:46,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,861 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:46,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,864 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,864 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:46,864 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:46,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:46,864 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:46,864 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:46,864 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:46,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:46,864 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,864 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:46,865 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:46,865 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:46,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:46,864 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:46,864 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,865 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,865 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:46,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:46,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:46,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042206926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:46,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042206926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:46,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042206926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:46,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042206926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:46,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042206926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,965 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:46,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:46,967 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:46,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:46,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:46,968 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:46,968 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:46,968 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:46,968 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:46,969 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,969 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:46,970 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,970 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:46,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:46,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:46,971 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,971 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:46,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:46,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:46,972 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:46,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:46,972 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:46,972 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:46,972 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:46,972 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:46,972 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:46,972 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:46,972 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:46,973 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:46,973 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:46,973 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:46,973 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:46,973 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:46,973 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,973 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:46,973 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:46,974 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,974 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:46,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:46,975 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,975 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:46,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:46,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:46,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:46,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:46,977 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:46,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27f7ebb2[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:46,977 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,978 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,980 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:46,980 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:46,980 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:46,980 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,980 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:46,981 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,981 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:46,981 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:46,981 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:46,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:46,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:46,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:46,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:46,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:46,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:46,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:46,984 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:46,984 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:46,984 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:46,984 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:46,984 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:46,986 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,986 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:46,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:46,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,986 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:46,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:46,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:46,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:46,987 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:46,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,988 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:46,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,991 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,991 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:46,991 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:46,991 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:46,991 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:46,991 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:46,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:46,991 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:46,991 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:46,991 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:46,991 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:46,991 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:46,991 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,991 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:46,991 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:46,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:47,091 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:47,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:47,094 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:47,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:47,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:47,100 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:47,100 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:47,100 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:47,100 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:47,100 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,100 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,102 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,102 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:47,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:47,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:47,102 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,103 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:47,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:47,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:47,104 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:47,104 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:47,104 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:47,104 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:47,104 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:47,104 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:47,104 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:47,104 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:47,104 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:47,104 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:47,104 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:47,105 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,105 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:47,105 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:47,106 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,106 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:47,106 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,106 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:47,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:47,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:47,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:47,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:47,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ae57142[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:47,108 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:47,108 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,109 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,109 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:47,109 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:47,109 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:47,111 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:47,111 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:47,111 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,111 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:47,111 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:47,111 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:47,111 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:47,111 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:47,111 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:47,111 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:47,111 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:47,114 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,114 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,114 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:47,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:47,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:47,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:47,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:47,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:47,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,115 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:47,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:47,116 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,118 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,118 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,118 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,119 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:47,119 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,119 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:47,119 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:47,119 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:47,119 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:47,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:47,119 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:47,119 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:47,119 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,119 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:47,119 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:47,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:47,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:47,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042207132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:47,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042207132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:47,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042207133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:47,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042207133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:47,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042207133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,218 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/485d02a215bc41489f4e33ab7194fd68 2023-04-21 01:55:47,219 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:47,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:47,222 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:47,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:47,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:47,223 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:47,223 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:47,223 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:47,223 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:47,224 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,224 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,225 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:47,225 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,226 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,226 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:47,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:47,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:47,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:47,228 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:47,228 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:47,228 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:47,238 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:47,238 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:47,238 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:47,238 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:47,238 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:47,238 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:47,238 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:47,238 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:47,239 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,239 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:47,239 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:47,242 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/01d328a167d6477db09ad7c116881e8f 2023-04-21 01:55:47,242 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:47,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:47,242 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:47,243 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,243 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:47,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:47,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:47,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:47,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@382e2697[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:47,245 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:47,245 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,246 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/09b09166d5ca41afaa29427354ac5934 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/09b09166d5ca41afaa29427354ac5934 2023-04-21 01:55:47,247 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,248 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:47,248 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,248 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:47,248 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:47,248 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:47,249 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,249 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:47,250 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:47,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:47,250 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,251 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/09b09166d5ca41afaa29427354ac5934, entries=200, sequenceid=95, filesize=13.9 K 2023-04-21 01:55:47,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,252 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/485d02a215bc41489f4e33ab7194fd68 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/485d02a215bc41489f4e33ab7194fd68 2023-04-21 01:55:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:47,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:47,254 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:47,254 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:47,254 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:47,254 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:47,254 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:47,255 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,256 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/485d02a215bc41489f4e33ab7194fd68, entries=150, sequenceid=95, filesize=11.6 K 2023-04-21 01:55:47,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,256 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:47,256 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:47,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:47,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:47,256 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/01d328a167d6477db09ad7c116881e8f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/01d328a167d6477db09ad7c116881e8f 2023-04-21 01:55:47,257 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,258 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,260 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/01d328a167d6477db09ad7c116881e8f, entries=150, sequenceid=95, filesize=11.6 K 2023-04-21 01:55:47,261 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:47,261 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:47,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:47,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:47,261 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,261 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,261 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,261 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:47,261 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:47,261 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,261 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,261 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:47,261 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,261 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,261 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:47,262 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:47,262 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=46.96 KB/48090 for 82ecbe07d2c67bb21d6485daedbb8c82 in 466ms, sequenceid=95, compaction requested=true 2023-04-21 01:55:47,262 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:47,262 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:47,262 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:55:47,262 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:55:47,262 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:47,262 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:47,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:47,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:47,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:47,264 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49880 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:55:47,264 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/A is initiating minor compaction (all files) 2023-04-21 01:55:47,264 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/A in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:47,264 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/0c7e3c7287fd45f58d3510b3713ff792, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/25c0a161746540f0bc6474b64ef5e060, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/fbf94c7344cb49e583bd4436e2030160, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/09b09166d5ca41afaa29427354ac5934] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=48.7 K 2023-04-21 01:55:47,264 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:55:47,264 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 0c7e3c7287fd45f58d3510b3713ff792, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682042145161 2023-04-21 01:55:47,264 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/B is initiating minor compaction (all files) 2023-04-21 01:55:47,265 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/B in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:47,265 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/885a1c3c1cf34c01bd6bad99d90ce34a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/55f5a950fec74790be7d8a2da0f0f1f1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/e729994638cd45a495e2b0c7774a8ad0, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/485d02a215bc41489f4e33ab7194fd68] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=44.1 K 2023-04-21 01:55:47,265 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 25c0a161746540f0bc6474b64ef5e060, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682042145200 2023-04-21 01:55:47,265 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 885a1c3c1cf34c01bd6bad99d90ce34a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682042145161 2023-04-21 01:55:47,265 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting fbf94c7344cb49e583bd4436e2030160, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682042145309 2023-04-21 01:55:47,265 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 55f5a950fec74790be7d8a2da0f0f1f1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682042145201 2023-04-21 01:55:47,265 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 09b09166d5ca41afaa29427354ac5934, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682042146477 2023-04-21 01:55:47,266 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting e729994638cd45a495e2b0c7774a8ad0, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682042145309 2023-04-21 01:55:47,267 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 485d02a215bc41489f4e33ab7194fd68, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682042146477 2023-04-21 01:55:47,276 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#A#compaction#426 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:47,278 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#B#compaction#427 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:47,362 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:47,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:47,364 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:47,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:47,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:47,365 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:47,365 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:47,365 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:47,365 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:47,366 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,366 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,368 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:47,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:47,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:47,368 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,368 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,368 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:47,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:47,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:47,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:47,370 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:47,370 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:47,370 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:47,370 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:47,370 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:47,370 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:47,370 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:47,371 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-21 01:55:47,371 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:47,371 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:47,371 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:47,371 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:47,371 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:47,371 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:47,390 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/fef85d2f8db048d9a301c553524d9b27 2023-04-21 01:55:47,410 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/6eb929e77fc34e16b30ac3ceb855c009 2023-04-21 01:55:47,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:47,441 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:47,446 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/efa5050cb5d147709445b7d833fa3ca6 2023-04-21 01:55:47,451 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/fef85d2f8db048d9a301c553524d9b27 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/fef85d2f8db048d9a301c553524d9b27 2023-04-21 01:55:47,455 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/fef85d2f8db048d9a301c553524d9b27, entries=150, sequenceid=105, filesize=11.6 K 2023-04-21 01:55:47,456 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/6eb929e77fc34e16b30ac3ceb855c009 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/6eb929e77fc34e16b30ac3ceb855c009 2023-04-21 01:55:47,461 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/6eb929e77fc34e16b30ac3ceb855c009, entries=150, sequenceid=105, filesize=11.6 K 2023-04-21 01:55:47,461 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/efa5050cb5d147709445b7d833fa3ca6 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/efa5050cb5d147709445b7d833fa3ca6 2023-04-21 01:55:47,466 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/efa5050cb5d147709445b7d833fa3ca6, entries=150, sequenceid=105, filesize=11.6 K 2023-04-21 01:55:47,466 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=93.93 KB/96180 for 82ecbe07d2c67bb21d6485daedbb8c82 in 95ms, sequenceid=105, compaction requested=true 2023-04-21 01:55:47,466 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:47,467 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:47,467 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:47,467 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:47,467 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,467 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:47,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:47,467 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-21 01:55:47,467 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:47,467 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:47,467 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:47,467 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:47,467 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:47,467 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:47,467 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:47,469 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:47,469 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:47,470 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,470 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:47,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:47,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:47,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:47,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dd118c5[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:47,471 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:47,472 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,473 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,473 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:47,473 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:47,473 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:47,473 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,473 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:47,475 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:47,475 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:47,475 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:47,475 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:47,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:47,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:47,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:47,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:47,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:47,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:47,477 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:47,478 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:47,478 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:47,478 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:47,478 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:47,484 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,484 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:47,484 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,485 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:47,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:47,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:47,486 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:47,487 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/cdfe0fbde7d946fca5fdbeb9907451ca 2023-04-21 01:55:47,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:47,488 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,488 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:47,488 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:47,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:47,489 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:47,488 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,489 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,489 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,489 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,489 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:47,488 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:47,489 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:47,489 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:47,489 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:47,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:47,489 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:47,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:47,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042207485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:47,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:47,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042207486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042207491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:47,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042207500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:47,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042207500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,590 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:47,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:47,592 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:47,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:47,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:47,593 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:47,593 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:47,593 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:47,593 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:47,594 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,594 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,595 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:47,595 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:47,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:47,595 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,595 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:47,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:47,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:47,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:47,597 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:47,597 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:47,597 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:47,598 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:47,598 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:47,598 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:47,598 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:47,598 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:47,598 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:47,598 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:47,598 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:47,598 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,598 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:47,598 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:47,600 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,600 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:47,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:47,600 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,600 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:47,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:47,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:47,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e3cab6[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:47,604 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:47,604 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:47,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042207601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:47,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042207601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:47,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042207601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,606 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,607 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:47,607 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:47,607 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:47,607 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,607 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:47,608 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:47,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:47,608 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:47,608 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:47,608 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:47,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:47,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:47,611 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:47,611 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:47,611 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:47,611 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:47,611 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:47,612 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,613 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,613 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:47,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:47,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:47,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042207606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042207607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,613 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,615 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:47,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:47,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:47,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:47,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:47,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,619 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:47,620 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:47,620 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:47,620 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:47,620 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:47,620 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:47,687 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/46f5cc40e3e84727bf45d2beeac7ed27 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/46f5cc40e3e84727bf45d2beeac7ed27 2023-04-21 01:55:47,697 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/A of 82ecbe07d2c67bb21d6485daedbb8c82 into 46f5cc40e3e84727bf45d2beeac7ed27(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:47,697 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:47,697 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/A, priority=12, startTime=1682042147262; duration=0sec 2023-04-21 01:55:47,697 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:47,697 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-21 01:55:47,699 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-21 01:55:47,699 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/C is initiating minor compaction (all files) 2023-04-21 01:55:47,699 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/C in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:47,699 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/4acfb3cc437f4afb8dddd95c05908554, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/b3ddb399295740d89551fc99e1cd618a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1e7ce63ba7ba4bda83b09afe66644e48, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/01d328a167d6477db09ad7c116881e8f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/efa5050cb5d147709445b7d833fa3ca6] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=55.7 K 2023-04-21 01:55:47,699 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 4acfb3cc437f4afb8dddd95c05908554, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682042145161 2023-04-21 01:55:47,700 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting b3ddb399295740d89551fc99e1cd618a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682042145201 2023-04-21 01:55:47,700 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 1e7ce63ba7ba4bda83b09afe66644e48, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682042145309 2023-04-21 01:55:47,700 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 01d328a167d6477db09ad7c116881e8f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682042146477 2023-04-21 01:55:47,700 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting efa5050cb5d147709445b7d833fa3ca6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1682042146797 2023-04-21 01:55:47,703 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/1586527042ae4a69a7a71ea681654140 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/1586527042ae4a69a7a71ea681654140 2023-04-21 01:55:47,709 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#C#compaction#433 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:47,710 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/B of 82ecbe07d2c67bb21d6485daedbb8c82 into 1586527042ae4a69a7a71ea681654140(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:47,710 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:47,710 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/B, priority=12, startTime=1682042147262; duration=0sec 2023-04-21 01:55:47,710 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:47,717 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/f2d9c80ac15e4cc9ad777d0e57f4451e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/f2d9c80ac15e4cc9ad777d0e57f4451e 2023-04-21 01:55:47,720 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:47,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:47,721 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/C of 82ecbe07d2c67bb21d6485daedbb8c82 into f2d9c80ac15e4cc9ad777d0e57f4451e(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:47,721 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:47,721 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/C, priority=11, startTime=1682042147262; duration=0sec 2023-04-21 01:55:47,721 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:47,722 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:47,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:47,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:47,724 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:47,724 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:47,724 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:47,724 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:47,725 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,725 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,727 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,727 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:47,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:47,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:47,727 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,727 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:47,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,728 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:47,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:47,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:47,728 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:47,728 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:47,728 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:47,729 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:47,729 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:47,730 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:47,730 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:47,730 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:47,730 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:47,730 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:47,730 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:47,730 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,730 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:47,730 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:47,732 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:47,733 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:47,733 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:47,733 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:47,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:47,734 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:47,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40dd429[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:47,734 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,735 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,735 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:47,736 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,736 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:47,736 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:47,736 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:47,737 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:47,737 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:47,737 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:47,737 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:47,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:47,740 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:47,740 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:47,740 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:47,740 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:47,741 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:47,743 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,743 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:47,743 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:47,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:47,743 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,746 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,748 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:47,748 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:47,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:47,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:47,748 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:47,748 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:47,748 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:47,748 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:47,748 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,748 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,749 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,749 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:47,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:47,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:47,749 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:47,749 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,749 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,749 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:47,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042207808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:47,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042207808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:47,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042207808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:47,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042207815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:47,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042207815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,849 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:47,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:47,851 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:47,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:47,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:47,852 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:47,852 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:47,852 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:47,852 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:47,853 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,853 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,854 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:47,854 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:47,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:47,854 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,854 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:47,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:47,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:47,856 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:47,856 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:47,857 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:47,857 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:47,857 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:47,857 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:47,857 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:47,857 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:47,857 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:47,857 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:47,857 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:47,857 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,857 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:47,857 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:47,859 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,859 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:47,859 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,859 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:47,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:47,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d5e50a1[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:47,861 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:47,861 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,863 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,863 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:47,863 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:47,863 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:47,864 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,864 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:47,865 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:47,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:47,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:47,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:47,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:47,870 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:47,870 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:47,870 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:47,870 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:47,871 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:47,872 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,872 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:47,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:47,872 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:47,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:47,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:47,873 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:47,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,874 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,880 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,880 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:47,880 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,880 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:47,880 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:47,880 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:47,880 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:47,880 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:47,881 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:47,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:47,880 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,881 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:47,881 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,881 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:47,881 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,895 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/a61ac1d250e34a84a3994364f46dd43b 2023-04-21 01:55:47,904 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/de556dc581424f3db593444a99d15e24 2023-04-21 01:55:47,908 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/cdfe0fbde7d946fca5fdbeb9907451ca as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/cdfe0fbde7d946fca5fdbeb9907451ca 2023-04-21 01:55:47,912 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/cdfe0fbde7d946fca5fdbeb9907451ca, entries=350, sequenceid=123, filesize=20.8 K 2023-04-21 01:55:47,913 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/a61ac1d250e34a84a3994364f46dd43b as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/a61ac1d250e34a84a3994364f46dd43b 2023-04-21 01:55:47,916 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/a61ac1d250e34a84a3994364f46dd43b, entries=150, sequenceid=123, filesize=11.6 K 2023-04-21 01:55:47,924 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/de556dc581424f3db593444a99d15e24 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/de556dc581424f3db593444a99d15e24 2023-04-21 01:55:47,927 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/de556dc581424f3db593444a99d15e24, entries=150, sequenceid=123, filesize=11.6 K 2023-04-21 01:55:47,928 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 82ecbe07d2c67bb21d6485daedbb8c82 in 461ms, sequenceid=123, compaction requested=true 2023-04-21 01:55:47,928 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:47,928 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:47,929 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:47,929 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:47,929 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:47,930 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:47,930 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:47,930 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/B is initiating minor compaction (all files) 2023-04-21 01:55:47,930 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/B in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:47,930 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/1586527042ae4a69a7a71ea681654140, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/6eb929e77fc34e16b30ac3ceb855c009, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/a61ac1d250e34a84a3994364f46dd43b] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=35.0 K 2023-04-21 01:55:47,931 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 45160 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:47,931 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 1586527042ae4a69a7a71ea681654140, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682042146477 2023-04-21 01:55:47,931 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/A is initiating minor compaction (all files) 2023-04-21 01:55:47,931 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/A in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:47,931 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/46f5cc40e3e84727bf45d2beeac7ed27, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/fef85d2f8db048d9a301c553524d9b27, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/cdfe0fbde7d946fca5fdbeb9907451ca] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=44.1 K 2023-04-21 01:55:47,931 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 6eb929e77fc34e16b30ac3ceb855c009, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1682042146797 2023-04-21 01:55:47,932 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 46f5cc40e3e84727bf45d2beeac7ed27, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682042146477 2023-04-21 01:55:47,932 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting a61ac1d250e34a84a3994364f46dd43b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682042147457 2023-04-21 01:55:47,933 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting fef85d2f8db048d9a301c553524d9b27, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1682042146797 2023-04-21 01:55:47,933 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting cdfe0fbde7d946fca5fdbeb9907451ca, keycount=350, bloomtype=ROW, size=20.8 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682042147447 2023-04-21 01:55:47,945 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#B#compaction#435 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:47,946 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#A#compaction#436 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:47,970 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/f21c0a3e4b2e45149c8a28200631c3ee as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/f21c0a3e4b2e45149c8a28200631c3ee 2023-04-21 01:55:47,975 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/A of 82ecbe07d2c67bb21d6485daedbb8c82 into f21c0a3e4b2e45149c8a28200631c3ee(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:47,975 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:47,975 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/A, priority=13, startTime=1682042147928; duration=0sec 2023-04-21 01:55:47,975 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:47,975 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-21 01:55:47,976 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-21 01:55:47,976 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-21 01:55:47,976 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. because compaction request was cancelled 2023-04-21 01:55:47,981 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:47,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:47,984 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:47,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:47,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:47,985 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:47,985 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:47,985 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:47,985 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:47,986 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,986 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,987 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,987 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:47,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:47,987 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:47,987 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:47,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:47,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:47,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:47,988 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:47,988 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:47,988 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:47,991 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:47,991 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:47,991 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:47,991 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:47,991 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-21 01:55:47,991 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:47,991 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:47,991 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:47,991 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:47,991 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:47,992 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:47,998 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/a1097efe5aea4696805c09254aeddd4b 2023-04-21 01:55:48,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:48,120 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:48,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042208135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042208136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042208137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042208138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042208138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042208242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042208244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042208245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042208246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042208246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,380 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/cd0e647d3c6640b19b8e0bb3833646f8 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/cd0e647d3c6640b19b8e0bb3833646f8 2023-04-21 01:55:48,384 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/B of 82ecbe07d2c67bb21d6485daedbb8c82 into cd0e647d3c6640b19b8e0bb3833646f8(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:48,385 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:48,385 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/B, priority=13, startTime=1682042147929; duration=0sec 2023-04-21 01:55:48,385 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:48,419 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/b4df412cb07046da9b1b883ac5e580da 2023-04-21 01:55:48,428 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/75a91c2fd54142849a5495a7568c0677 2023-04-21 01:55:48,433 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/a1097efe5aea4696805c09254aeddd4b as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/a1097efe5aea4696805c09254aeddd4b 2023-04-21 01:55:48,437 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/a1097efe5aea4696805c09254aeddd4b, entries=150, sequenceid=142, filesize=11.8 K 2023-04-21 01:55:48,438 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/b4df412cb07046da9b1b883ac5e580da as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/b4df412cb07046da9b1b883ac5e580da 2023-04-21 01:55:48,441 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/b4df412cb07046da9b1b883ac5e580da, entries=150, sequenceid=142, filesize=11.8 K 2023-04-21 01:55:48,441 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/75a91c2fd54142849a5495a7568c0677 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/75a91c2fd54142849a5495a7568c0677 2023-04-21 01:55:48,446 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/75a91c2fd54142849a5495a7568c0677, entries=150, sequenceid=142, filesize=11.8 K 2023-04-21 01:55:48,446 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 82ecbe07d2c67bb21d6485daedbb8c82 in 455ms, sequenceid=142, compaction requested=true 2023-04-21 01:55:48,446 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:48,447 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:48,447 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:48,447 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:48,447 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,447 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:48,447 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:48,448 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,448 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:48,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:48,449 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,449 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:48,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:48,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:48,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:48,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:48,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:48,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58dc7eff[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:48,450 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:48,450 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:48,451 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-21 01:55:48,451 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:48,452 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:48,452 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:48,452 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:48,452 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:48,452 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:48,452 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,452 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:48,452 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:48,452 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:48,453 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,453 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:48,455 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:48,455 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:48,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,456 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:48,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:48,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:48,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:48,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:48,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:48,459 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:48,459 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:48,459 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:48,459 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:48,459 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:48,461 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,461 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:48,461 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:48,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:48,462 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:48,464 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:48,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:48,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:48,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:48,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:48,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:48,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,466 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:48,466 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:48,466 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:48,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:48,466 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:48,466 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:48,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:48,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042208469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042208478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042208498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042208498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042208498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,567 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:48,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:48,569 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:48,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:48,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:48,570 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:48,570 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:48,570 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:48,570 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:48,571 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,571 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:48,572 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,572 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:48,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:48,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:48,572 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,572 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:48,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:48,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:48,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:48,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:48,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:48,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:48,574 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:48,574 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:48,574 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:48,574 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:48,574 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:48,574 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:48,574 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:48,574 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:48,574 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:48,574 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:48,574 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:48,574 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,574 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:48,574 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:48,576 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,576 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:48,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:48,577 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,577 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:48,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:48,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:48,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:48,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:48,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:48,578 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:48,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43f1a928[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:48,578 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,579 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,579 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:48,579 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:48,579 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:48,580 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,580 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:48,581 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:48,581 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:48,581 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,581 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:48,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:48,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:48,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:48,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:48,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:48,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:48,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:48,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:48,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:48,584 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:48,584 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:48,584 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:48,584 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:48,584 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:48,586 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,586 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:48,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:48,586 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:48,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:48,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:48,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:48,587 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:48,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:48,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:48,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,588 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:48,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:48,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,591 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:48,591 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:48,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:48,592 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:48,591 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:48,592 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:48,592 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:48,592 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:48,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:48,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:48,592 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:48,592 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:48,592 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:48,592 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,592 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,592 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,592 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042208599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042208604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042208605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042208605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042208605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,692 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:48,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:48,694 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:48,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:48,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:48,696 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:48,696 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:48,696 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:48,696 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:48,696 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,696 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:48,697 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,697 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:48,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:48,698 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,698 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:48,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:48,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:48,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:48,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:48,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:48,699 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:48,699 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:48,699 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:48,699 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:48,699 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:48,699 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:48,699 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:48,699 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:48,699 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:48,700 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:48,700 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:48,700 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,700 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:48,700 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:48,701 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,701 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:48,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:48,701 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,701 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:48,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:48,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:48,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:48,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:48,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:48,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@293791e3[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:48,703 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:48,703 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,704 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,704 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:48,704 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:48,704 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:48,704 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,705 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:48,707 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:48,707 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:48,707 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,707 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:48,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:48,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:48,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:48,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:48,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:48,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:48,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:48,710 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:48,710 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:48,710 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:48,710 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:48,710 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:48,711 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,711 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:48,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:48,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:48,712 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:48,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:48,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:48,713 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:48,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:48,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:48,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,713 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:48,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:48,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,718 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,718 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:48,718 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:48,718 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:48,718 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:48,718 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:48,718 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:48,720 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:48,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:48,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:48,718 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,720 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,720 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:48,720 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,720 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,720 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:48,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:48,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:48,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042208803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042208810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042208810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042208810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,817 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:48,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042208811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,820 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:48,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:48,822 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:48,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:48,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:48,824 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:48,824 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:48,824 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:48,824 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:48,824 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,824 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:48,826 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:48,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:48,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:48,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:48,827 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:48,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:48,827 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,827 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:48,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:48,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:48,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:48,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:48,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:48,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:48,829 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:48,829 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:48,829 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:48,829 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:48,830 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:48,830 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:48,830 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:48,830 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:48,830 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,830 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:48,830 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:48,831 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,831 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:48,831 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,831 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:48,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:48,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:48,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:48,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:48,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:48,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1653173c[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:48,835 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:48,835 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,836 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,836 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:48,836 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,836 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:48,836 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:48,836 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:48,837 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:48,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:48,840 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:48,840 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:48,840 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:48,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:48,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:48,841 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:48,841 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:48,841 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:48,841 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:48,842 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:48,844 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,844 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:48,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,844 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:48,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:48,845 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,846 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:48,848 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:48,848 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:48,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:48,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:48,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:48,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:48,850 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,850 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:48,850 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:48,850 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:48,850 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:48,850 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:48,850 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,850 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,850 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:48,850 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:48,850 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:48,850 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:48,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:48,862 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/174f50c0245e466685c59a0e3136196b 2023-04-21 01:55:48,877 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/2813bb7c354f4daab02555fb4ad06472 2023-04-21 01:55:48,890 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/1619b26e46d442e9a43de6e1cb5a1044 2023-04-21 01:55:48,894 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/174f50c0245e466685c59a0e3136196b as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/174f50c0245e466685c59a0e3136196b 2023-04-21 01:55:48,897 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/174f50c0245e466685c59a0e3136196b, entries=250, sequenceid=162, filesize=16.4 K 2023-04-21 01:55:48,898 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/2813bb7c354f4daab02555fb4ad06472 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/2813bb7c354f4daab02555fb4ad06472 2023-04-21 01:55:48,901 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/2813bb7c354f4daab02555fb4ad06472, entries=150, sequenceid=162, filesize=11.8 K 2023-04-21 01:55:48,903 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/1619b26e46d442e9a43de6e1cb5a1044 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1619b26e46d442e9a43de6e1cb5a1044 2023-04-21 01:55:48,906 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1619b26e46d442e9a43de6e1cb5a1044, entries=150, sequenceid=162, filesize=11.8 K 2023-04-21 01:55:48,906 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=80.51 KB/82440 for 82ecbe07d2c67bb21d6485daedbb8c82 in 455ms, sequenceid=162, compaction requested=true 2023-04-21 01:55:48,906 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:48,907 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:48,907 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:48,907 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:48,907 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:48,907 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:48,908 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40978 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:48,908 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36198 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:48,908 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/A is initiating minor compaction (all files) 2023-04-21 01:55:48,908 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/B is initiating minor compaction (all files) 2023-04-21 01:55:48,908 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/A in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:48,908 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/B in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:48,908 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/f21c0a3e4b2e45149c8a28200631c3ee, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/a1097efe5aea4696805c09254aeddd4b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/174f50c0245e466685c59a0e3136196b] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=40.0 K 2023-04-21 01:55:48,908 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/cd0e647d3c6640b19b8e0bb3833646f8, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/b4df412cb07046da9b1b883ac5e580da, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/2813bb7c354f4daab02555fb4ad06472] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=35.3 K 2023-04-21 01:55:48,909 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting f21c0a3e4b2e45149c8a28200631c3ee, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682042147457 2023-04-21 01:55:48,909 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting cd0e647d3c6640b19b8e0bb3833646f8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682042147457 2023-04-21 01:55:48,909 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting a1097efe5aea4696805c09254aeddd4b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682042147484 2023-04-21 01:55:48,909 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting b4df412cb07046da9b1b883ac5e580da, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682042147484 2023-04-21 01:55:48,909 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 174f50c0245e466685c59a0e3136196b, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682042148134 2023-04-21 01:55:48,910 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 2813bb7c354f4daab02555fb4ad06472, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682042148137 2023-04-21 01:55:48,915 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#B#compaction#444 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:48,916 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#A#compaction#443 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:48,930 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/0f938e9503a642429449ff8a7d80901b as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/0f938e9503a642429449ff8a7d80901b 2023-04-21 01:55:48,931 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/97cdfc17065f466f962dd59609ad2897 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/97cdfc17065f466f962dd59609ad2897 2023-04-21 01:55:48,938 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/A of 82ecbe07d2c67bb21d6485daedbb8c82 into 97cdfc17065f466f962dd59609ad2897(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:48,938 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/B of 82ecbe07d2c67bb21d6485daedbb8c82 into 0f938e9503a642429449ff8a7d80901b(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:48,938 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:48,939 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/A, priority=13, startTime=1682042148907; duration=0sec 2023-04-21 01:55:48,939 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:48,939 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:55:48,938 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:48,940 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/B, priority=13, startTime=1682042148907; duration=0sec 2023-04-21 01:55:48,940 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:48,941 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48016 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:55:48,941 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/C is initiating minor compaction (all files) 2023-04-21 01:55:48,941 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/C in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:48,941 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/f2d9c80ac15e4cc9ad777d0e57f4451e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/de556dc581424f3db593444a99d15e24, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/75a91c2fd54142849a5495a7568c0677, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1619b26e46d442e9a43de6e1cb5a1044] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=46.9 K 2023-04-21 01:55:48,941 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting f2d9c80ac15e4cc9ad777d0e57f4451e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1682042146797 2023-04-21 01:55:48,942 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting de556dc581424f3db593444a99d15e24, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682042147457 2023-04-21 01:55:48,942 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 75a91c2fd54142849a5495a7568c0677, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682042147484 2023-04-21 01:55:48,942 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 1619b26e46d442e9a43de6e1cb5a1044, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682042148137 2023-04-21 01:55:48,950 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#C#compaction#445 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:48,951 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:48,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:48,955 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:48,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:48,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:48,958 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:48,958 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:48,959 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:48,959 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:48,959 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,959 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:48,960 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:48,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:48,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:48,960 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:48,961 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:48,961 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:48,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:48,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:48,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:48,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:48,963 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:48,963 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:48,963 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:48,964 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:48,964 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:48,964 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:48,964 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:48,964 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-21 01:55:48,964 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:48,965 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:48,965 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:48,965 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:48,965 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:48,965 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:48,968 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/1aca0bc1f6484bccb0c4ce911e4d2a8e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1aca0bc1f6484bccb0c4ce911e4d2a8e 2023-04-21 01:55:48,971 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/C of 82ecbe07d2c67bb21d6485daedbb8c82 into 1aca0bc1f6484bccb0c4ce911e4d2a8e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:48,971 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:48,972 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/C, priority=12, startTime=1682042148907; duration=0sec 2023-04-21 01:55:48,972 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:48,985 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/3070dd2fdaf44bc39fc20e81fd9b7d1e 2023-04-21 01:55:49,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:49,113 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:49,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:49,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042209125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:49,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042209126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:49,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042209127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:49,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042209128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:49,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042209131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:49,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042209232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:49,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042209233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:49,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042209234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:49,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042209234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:49,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042209239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,404 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/a9e42a8f89374f448ccd47a580434f9a 2023-04-21 01:55:49,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:49,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042209438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:49,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042209438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:49,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042209439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:49,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042209440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:49,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042209444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,612 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-21 01:55:49,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:49,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042209743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:49,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042209743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:49,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042209744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,749 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:49,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042209746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:49,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042209748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,818 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/c5019b55068b4d39919aa065f3e0e3ba 2023-04-21 01:55:49,822 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/3070dd2fdaf44bc39fc20e81fd9b7d1e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/3070dd2fdaf44bc39fc20e81fd9b7d1e 2023-04-21 01:55:49,825 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/3070dd2fdaf44bc39fc20e81fd9b7d1e, entries=150, sequenceid=179, filesize=11.8 K 2023-04-21 01:55:49,825 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/a9e42a8f89374f448ccd47a580434f9a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/a9e42a8f89374f448ccd47a580434f9a 2023-04-21 01:55:49,833 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/a9e42a8f89374f448ccd47a580434f9a, entries=150, sequenceid=179, filesize=11.8 K 2023-04-21 01:55:49,834 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/c5019b55068b4d39919aa065f3e0e3ba as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/c5019b55068b4d39919aa065f3e0e3ba 2023-04-21 01:55:49,838 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/c5019b55068b4d39919aa065f3e0e3ba, entries=150, sequenceid=179, filesize=11.8 K 2023-04-21 01:55:49,839 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 82ecbe07d2c67bb21d6485daedbb8c82 in 875ms, sequenceid=179, compaction requested=false 2023-04-21 01:55:49,839 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:49,839 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:49,839 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:49,839 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:49,839 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,839 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:49,839 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:49,840 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,841 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:49,841 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:49,841 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:49,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:49,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:49,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d43a88f[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:49,842 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:49,843 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:49,845 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:49,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:49,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:49,845 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,845 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:49,845 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:49,845 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:49,845 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:49,846 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:49,846 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:49,846 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,846 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:49,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:49,849 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:49,849 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:49,849 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:49,849 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:49,849 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:49,850 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:49,850 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:49,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:49,851 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:49,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:49,851 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:49,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:49,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:49,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:49,853 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:49,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:49,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,855 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:49,855 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,855 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:49,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:49,855 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:49,855 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:49,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:49,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:49,855 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:49,855 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:49,855 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:49,855 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,855 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:49,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:49,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:49,855 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:49,855 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:49,855 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:49,856 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:49,956 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:49,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:49,958 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:49,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:49,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:49,959 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:49,959 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:49,959 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:49,959 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:49,960 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:49,960 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:49,962 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:49,962 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:49,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:49,962 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:49,962 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:49,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:49,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:49,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:49,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:49,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:49,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:49,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:49,963 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:49,963 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:49,963 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:49,963 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:49,963 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:49,964 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:49,964 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:49,964 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-21 01:55:49,964 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:49,964 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:49,964 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:49,964 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:49,964 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:49,964 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:49,971 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/a60803fc98fb4717959010559c9f748d 2023-04-21 01:55:50,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:50,255 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:50,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:50,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042210265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:50,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042210266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:50,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:50,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042210267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:50,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042210268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042210267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:50,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042210369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,381 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/23faa52d32f3400a9c85203487eb40d7 2023-04-21 01:55:50,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:50,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042210374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:50,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042210374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:50,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042210374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:50,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042210374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:50,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042210574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:50,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042210583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:50,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042210584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:50,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042210584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:50,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042210584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,791 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/2696471ddbb64323a4764baa3fa63200 2023-04-21 01:55:50,795 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/a60803fc98fb4717959010559c9f748d as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/a60803fc98fb4717959010559c9f748d 2023-04-21 01:55:50,799 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/a60803fc98fb4717959010559c9f748d, entries=150, sequenceid=197, filesize=11.8 K 2023-04-21 01:55:50,799 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/23faa52d32f3400a9c85203487eb40d7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/23faa52d32f3400a9c85203487eb40d7 2023-04-21 01:55:50,803 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/23faa52d32f3400a9c85203487eb40d7, entries=150, sequenceid=197, filesize=11.8 K 2023-04-21 01:55:50,804 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/2696471ddbb64323a4764baa3fa63200 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/2696471ddbb64323a4764baa3fa63200 2023-04-21 01:55:50,807 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/2696471ddbb64323a4764baa3fa63200, entries=150, sequenceid=197, filesize=11.8 K 2023-04-21 01:55:50,808 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 82ecbe07d2c67bb21d6485daedbb8c82 in 844ms, sequenceid=197, compaction requested=true 2023-04-21 01:55:50,808 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:50,808 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:50,808 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:50,808 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:50,808 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,808 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:50,808 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:50,810 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,810 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:50,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:50,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:50,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:50,810 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:50,810 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:50,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:50,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:50,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:50,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:50,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6335ece6[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:50,812 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:50,812 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:50,813 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:50,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:50,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:50,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:50,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:50,813 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,813 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:50,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:50,814 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,814 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:50,815 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:50,815 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:50,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:50,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:50,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:50,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:50,817 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:50,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:50,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:50,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:50,817 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:50,819 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:50,819 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:50,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:50,819 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:50,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:50,820 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:50,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:50,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:50,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,822 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:50,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:50,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,824 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,824 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:50,824 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:50,824 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,824 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:50,824 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:50,824 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:50,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:50,824 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:50,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:50,824 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:50,824 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:50,824 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:50,824 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:50,824 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:50,824 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:50,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:50,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:50,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:50,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:50,885 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-21 01:55:50,886 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:50,886 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:50,886 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:50,886 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:50,886 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:50,886 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:50,894 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/2d6caea96b0c413bb739ce4e07a4db6b 2023-04-21 01:55:50,903 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/5e0bcad35ed44556b0230f352d3d6860 2023-04-21 01:55:50,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:50,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042210906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:50,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042210906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,914 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/8bc105bf3fbb40858bde5153e6092b28 2023-04-21 01:55:50,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:50,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042210907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:50,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042210907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,917 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/2d6caea96b0c413bb739ce4e07a4db6b as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/2d6caea96b0c413bb739ce4e07a4db6b 2023-04-21 01:55:50,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:50,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042210914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,924 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/2d6caea96b0c413bb739ce4e07a4db6b, entries=200, sequenceid=213, filesize=14.1 K 2023-04-21 01:55:50,925 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:50,925 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/5e0bcad35ed44556b0230f352d3d6860 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/5e0bcad35ed44556b0230f352d3d6860 2023-04-21 01:55:50,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:50,927 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:50,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:50,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:50,928 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:50,928 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:50,928 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:50,928 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:50,929 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:50,929 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:50,929 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/5e0bcad35ed44556b0230f352d3d6860, entries=150, sequenceid=213, filesize=11.8 K 2023-04-21 01:55:50,930 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/8bc105bf3fbb40858bde5153e6092b28 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/8bc105bf3fbb40858bde5153e6092b28 2023-04-21 01:55:50,930 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:50,930 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:50,931 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,931 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:50,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:50,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:50,932 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:50,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:50,932 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:50,932 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:50,932 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:50,932 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:50,932 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:50,932 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:50,932 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:50,932 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:50,932 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:50,932 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:50,932 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:50,932 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,933 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:50,933 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:50,937 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/8bc105bf3fbb40858bde5153e6092b28, entries=150, sequenceid=213, filesize=11.8 K 2023-04-21 01:55:50,938 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,938 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:50,938 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 82ecbe07d2c67bb21d6485daedbb8c82 in 53ms, sequenceid=213, compaction requested=true 2023-04-21 01:55:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,938 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:50,938 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:50,938 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:50,938 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:55:50,938 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:50,938 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:55:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:50,938 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:50,938 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:50,939 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48486 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:55:50,939 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/B is initiating minor compaction (all files) 2023-04-21 01:55:50,940 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/B in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:50,940 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/0f938e9503a642429449ff8a7d80901b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/a9e42a8f89374f448ccd47a580434f9a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/23faa52d32f3400a9c85203487eb40d7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/5e0bcad35ed44556b0230f352d3d6860] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=47.3 K 2023-04-21 01:55:50,940 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50876 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:55:50,940 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/A is initiating minor compaction (all files) 2023-04-21 01:55:50,940 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/A in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:50,940 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/97cdfc17065f466f962dd59609ad2897, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/3070dd2fdaf44bc39fc20e81fd9b7d1e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/a60803fc98fb4717959010559c9f748d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/2d6caea96b0c413bb739ce4e07a4db6b] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=49.7 K 2023-04-21 01:55:50,940 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 0f938e9503a642429449ff8a7d80901b, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682042148137 2023-04-21 01:55:50,940 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 97cdfc17065f466f962dd59609ad2897, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682042148137 2023-04-21 01:55:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,941 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting a9e42a8f89374f448ccd47a580434f9a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1682042148468 2023-04-21 01:55:50,941 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 3070dd2fdaf44bc39fc20e81fd9b7d1e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1682042148468 2023-04-21 01:55:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:50,941 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 23faa52d32f3400a9c85203487eb40d7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1682042149125 2023-04-21 01:55:50,941 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting a60803fc98fb4717959010559c9f748d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1682042149125 2023-04-21 01:55:50,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:50,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7618d1ea[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:50,941 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:50,942 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:50,942 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 5e0bcad35ed44556b0230f352d3d6860, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682042150264 2023-04-21 01:55:50,942 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 2d6caea96b0c413bb739ce4e07a4db6b, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682042150264 2023-04-21 01:55:50,943 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:50,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:50,943 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:50,943 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,943 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:50,943 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:50,943 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:50,945 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:50,945 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:50,945 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:50,945 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:50,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:50,949 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:50,949 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:50,949 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:50,949 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:50,949 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:50,951 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:50,951 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:50,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:50,952 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:50,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:50,954 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:50,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:50,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:50,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:50,955 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,958 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#A#compaction#455 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:50,959 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:50,960 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:50,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:50,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:50,961 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,961 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:50,961 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:50,961 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:50,961 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:50,961 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:50,961 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:50,961 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:50,961 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:50,961 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:50,961 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:50,961 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:50,963 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#B#compaction#456 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:50,976 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/581948ab03c548caa931805755ae847a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/581948ab03c548caa931805755ae847a 2023-04-21 01:55:50,984 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/A of 82ecbe07d2c67bb21d6485daedbb8c82 into 581948ab03c548caa931805755ae847a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:50,984 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:50,984 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/A, priority=12, startTime=1682042150938; duration=0sec 2023-04-21 01:55:50,984 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/deb34bf218ef4281adc0660c94c0e93c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/deb34bf218ef4281adc0660c94c0e93c 2023-04-21 01:55:50,984 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:50,984 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:55:50,986 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:55:50,986 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/C is initiating minor compaction (all files) 2023-04-21 01:55:50,986 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/C in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:50,986 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1aca0bc1f6484bccb0c4ce911e4d2a8e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/c5019b55068b4d39919aa065f3e0e3ba, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/2696471ddbb64323a4764baa3fa63200, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/8bc105bf3fbb40858bde5153e6092b28] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=47.3 K 2023-04-21 01:55:50,987 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 1aca0bc1f6484bccb0c4ce911e4d2a8e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682042148137 2023-04-21 01:55:50,988 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting c5019b55068b4d39919aa065f3e0e3ba, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1682042148468 2023-04-21 01:55:50,988 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 2696471ddbb64323a4764baa3fa63200, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1682042149125 2023-04-21 01:55:50,988 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 8bc105bf3fbb40858bde5153e6092b28, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682042150264 2023-04-21 01:55:50,990 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/B of 82ecbe07d2c67bb21d6485daedbb8c82 into deb34bf218ef4281adc0660c94c0e93c(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:50,990 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:50,990 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/B, priority=12, startTime=1682042150938; duration=0sec 2023-04-21 01:55:50,990 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:50,995 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#C#compaction#457 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:51,005 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/9e7c9b829ee549208408c12a7ee3907a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/9e7c9b829ee549208408c12a7ee3907a 2023-04-21 01:55:51,010 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/C of 82ecbe07d2c67bb21d6485daedbb8c82 into 9e7c9b829ee549208408c12a7ee3907a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:51,010 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:51,010 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/C, priority=12, startTime=1682042150938; duration=0sec 2023-04-21 01:55:51,010 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:51,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:51,018 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-21 01:55:51,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:51,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:51,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:51,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:51,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:51,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:51,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042211030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042211030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042211031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042211032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042211035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,061 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:51,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:51,063 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:51,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:51,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:51,064 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:51,064 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:51,065 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:51,065 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:51,065 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,065 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:51,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,066 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:51,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:51,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:51,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:51,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:51,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:51,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:51,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:51,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:51,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:51,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:51,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:51,068 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:51,068 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:51,068 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:51,068 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:51,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:51,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:51,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:51,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:51,070 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,070 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:51,070 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,070 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:51,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:51,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:51,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:51,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:51,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78fcd2e2[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:51,072 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:51,072 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,073 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:51,074 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:51,074 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:51,074 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,074 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:51,076 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:51,076 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,076 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:51,076 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:51,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:51,081 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:51,081 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:51,081 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:51,081 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:51,081 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:51,083 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,083 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,083 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:51,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:51,084 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,085 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:51,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:51,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,087 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:51,087 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:51,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:51,087 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:51,087 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:51,087 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:51,087 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:51,088 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:51,088 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,088 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,088 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,088 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,087 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:51,088 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:51,088 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:51,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:51,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042211136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042211136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042211137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042211138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042211138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,188 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:51,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:51,190 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:51,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:51,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:51,191 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:51,191 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:51,191 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:51,191 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:51,192 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,192 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:51,193 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:51,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:51,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:51,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:51,194 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:51,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:51,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:51,195 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:51,195 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:51,195 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:51,195 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:51,195 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:51,195 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:51,195 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:51,195 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:51,195 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:51,195 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:51,195 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:51,195 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,195 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:51,195 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:51,197 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,197 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:51,198 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:51,198 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:51,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:51,199 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:51,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@484b1aa2[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:51,199 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,200 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,200 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:51,200 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:51,200 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:51,201 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,201 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:51,204 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,204 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:51,204 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:51,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:51,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:51,204 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:51,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:51,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:51,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:51,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:51,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:51,206 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:51,207 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:51,207 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:51,207 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:51,207 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:51,208 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,208 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,208 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:51,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:51,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:51,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:51,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:51,209 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,210 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:51,215 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,215 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:51,215 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:51,215 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:51,215 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:51,215 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:51,215 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:51,215 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:51,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:51,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:51,215 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:51,215 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:51,215 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:51,215 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,215 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,215 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:51,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:51,315 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:51,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:51,317 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:51,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:51,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:51,320 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:51,320 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:51,320 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:51,320 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:51,320 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,320 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:51,321 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:51,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:51,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:51,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:51,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:51,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:51,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:51,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:51,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:51,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:51,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:51,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:51,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:51,323 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:51,323 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:51,323 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:51,323 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:51,323 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:51,324 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:51,324 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,324 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:51,324 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:51,325 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,325 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:51,326 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,326 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:51,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:51,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d59d91[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:51,327 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:51,327 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,332 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,332 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:51,332 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:51,332 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:51,333 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:51,333 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,333 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:51,333 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:51,333 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:51,334 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:51,334 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:51,334 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:51,334 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:51,334 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:51,334 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:51,335 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,335 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:51,335 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:51,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,336 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:51,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:51,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:51,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,337 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,339 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:51,339 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:51,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:51,339 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:51,339 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:51,340 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:51,339 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:51,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:51,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:51,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:51,340 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:51,339 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:51,340 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:51,340 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:51,340 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,340 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,340 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,340 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,343 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042211342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,343 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042211342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042211342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042211342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042211343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,425 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/3b76d5f9c6914d319cdc147700c927f2 2023-04-21 01:55:51,438 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/eba2941809924d38a66a050fa808718f 2023-04-21 01:55:51,440 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:51,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:51,444 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:51,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:51,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:51,445 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:51,445 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:51,445 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:51,445 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:51,446 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,446 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:51,448 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/543e89f6686c4fbbb587e874c01c764b 2023-04-21 01:55:51,448 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:51,448 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:51,449 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:51,449 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:51,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:51,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:51,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:51,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:51,451 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:51,451 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:51,451 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:51,453 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/3b76d5f9c6914d319cdc147700c927f2 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/3b76d5f9c6914d319cdc147700c927f2 2023-04-21 01:55:51,456 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:51,456 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:51,456 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:51,456 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:51,456 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:51,457 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:51,457 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:51,457 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:51,457 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,457 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:51,457 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/3b76d5f9c6914d319cdc147700c927f2, entries=200, sequenceid=233, filesize=14.1 K 2023-04-21 01:55:51,457 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:51,457 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/eba2941809924d38a66a050fa808718f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/eba2941809924d38a66a050fa808718f 2023-04-21 01:55:51,458 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,458 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:51,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:51,459 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,459 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:51,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:51,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34729b60[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:51,461 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:51,461 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,464 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,464 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:51,464 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:51,464 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:51,464 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,464 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:51,464 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/eba2941809924d38a66a050fa808718f, entries=150, sequenceid=233, filesize=11.8 K 2023-04-21 01:55:51,465 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/543e89f6686c4fbbb587e874c01c764b as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/543e89f6686c4fbbb587e874c01c764b 2023-04-21 01:55:51,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,465 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:51,465 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:51,465 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:51,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:51,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:51,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:51,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:51,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:51,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:51,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:51,468 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:51,468 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:51,468 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:51,468 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:51,468 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:51,469 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/543e89f6686c4fbbb587e874c01c764b, entries=150, sequenceid=233, filesize=11.8 K 2023-04-21 01:55:51,470 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 82ecbe07d2c67bb21d6485daedbb8c82 in 452ms, sequenceid=233, compaction requested=false 2023-04-21 01:55:51,470 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:51,471 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,471 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:51,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,471 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:51,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:51,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:51,472 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:51,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:51,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:51,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:51,473 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:51,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:51,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:51,475 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,475 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:51,475 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:51,475 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:51,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:51,475 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:51,475 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:51,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:51,475 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,475 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:51,476 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:51,475 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,476 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:51,476 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:51,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:51,476 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:51,476 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:51,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:51,576 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:51,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:51,578 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:51,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:51,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:51,579 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:51,579 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:51,580 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:51,580 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:51,580 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,580 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:51,581 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:51,581 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:51,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:51,583 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,583 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:51,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:51,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:51,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:51,585 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:51,585 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:51,585 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:51,585 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:51,585 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:51,586 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:51,586 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:51,586 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:51,586 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:51,586 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-21 01:55:51,586 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:51,586 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:51,586 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:51,586 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:51,586 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:51,586 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:51,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:51,648 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:51,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042211659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042211662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042211663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042211663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042211664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042211765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042211765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042211770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042211772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042211772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042211976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042211976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042211976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,982 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042211980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:51,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042211980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:51,996 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/fee29d68ff884884bfc44d7029fbe390 2023-04-21 01:55:52,012 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/c4caf60449bc4610bcc89b20a3bee837 2023-04-21 01:55:52,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:52,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042212281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,291 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:52,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042212282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,291 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:52,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042212282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,291 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:52,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042212285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,291 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:52,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042212286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,431 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/d892e94d92054b8ba87da132d233373c 2023-04-21 01:55:52,434 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/fee29d68ff884884bfc44d7029fbe390 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/fee29d68ff884884bfc44d7029fbe390 2023-04-21 01:55:52,438 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/fee29d68ff884884bfc44d7029fbe390, entries=150, sequenceid=248, filesize=11.8 K 2023-04-21 01:55:52,439 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/c4caf60449bc4610bcc89b20a3bee837 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/c4caf60449bc4610bcc89b20a3bee837 2023-04-21 01:55:52,442 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/c4caf60449bc4610bcc89b20a3bee837, entries=150, sequenceid=248, filesize=11.8 K 2023-04-21 01:55:52,443 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/d892e94d92054b8ba87da132d233373c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/d892e94d92054b8ba87da132d233373c 2023-04-21 01:55:52,446 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/d892e94d92054b8ba87da132d233373c, entries=150, sequenceid=248, filesize=11.8 K 2023-04-21 01:55:52,447 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 82ecbe07d2c67bb21d6485daedbb8c82 in 861ms, sequenceid=248, compaction requested=true 2023-04-21 01:55:52,447 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:52,447 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:52,447 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:52,447 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:52,447 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,447 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:52,447 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:52,449 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:52,449 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:52,449 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:52,450 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:52,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:52,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a0724ce[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:52,452 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:52,452 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:52,454 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:52,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:52,455 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:52,455 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:52,455 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:52,455 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,455 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:52,456 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:52,456 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:52,456 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,456 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:52,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:52,459 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:52,459 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:52,459 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:52,459 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:52,459 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:52,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:52,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:52,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:52,460 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:52,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:52,461 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:52,462 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:52,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:52,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:52,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:52,465 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:52,465 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:52,465 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:52,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:52,465 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:52,465 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:52,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:52,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:52,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:52,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:52,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:52,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:52,565 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:52,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:52,568 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:52,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:52,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:52,569 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:52,569 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:52,569 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:52,569 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:52,570 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:52,570 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:52,573 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:52,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:52,574 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:52,574 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,574 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:52,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:52,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:52,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:52,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:52,575 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:52,575 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:52,575 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:52,575 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:52,575 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:52,575 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:52,575 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:52,575 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-21 01:55:52,575 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:52,575 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:52,576 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:52,576 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:52,576 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:52,576 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:52,585 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/127dbbf28cf84e3fb4fc1a8a1eda1ec4 2023-04-21 01:55:52,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:52,793 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:52,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:52,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042212807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:52,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042212809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:52,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042212810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:52,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042212811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:52,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042212812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:52,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042212915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:52,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042212915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:52,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042212919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:52,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042212919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:52,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042212919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:52,994 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/31e0bba869a74c609f528163607a31d6 2023-04-21 01:55:53,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:53,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042213119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:53,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042213120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:53,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042213128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:53,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042213128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:53,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042213130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,405 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/72f60ab3a79f41c79e4d743efbc27619 2023-04-21 01:55:53,409 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/127dbbf28cf84e3fb4fc1a8a1eda1ec4 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/127dbbf28cf84e3fb4fc1a8a1eda1ec4 2023-04-21 01:55:53,412 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/127dbbf28cf84e3fb4fc1a8a1eda1ec4, entries=150, sequenceid=265, filesize=11.9 K 2023-04-21 01:55:53,413 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/31e0bba869a74c609f528163607a31d6 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/31e0bba869a74c609f528163607a31d6 2023-04-21 01:55:53,417 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/31e0bba869a74c609f528163607a31d6, entries=150, sequenceid=265, filesize=11.9 K 2023-04-21 01:55:53,417 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/72f60ab3a79f41c79e4d743efbc27619 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/72f60ab3a79f41c79e4d743efbc27619 2023-04-21 01:55:53,421 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/72f60ab3a79f41c79e4d743efbc27619, entries=150, sequenceid=265, filesize=11.9 K 2023-04-21 01:55:53,422 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 82ecbe07d2c67bb21d6485daedbb8c82 in 847ms, sequenceid=265, compaction requested=true 2023-04-21 01:55:53,422 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:53,422 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:53,422 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:53,422 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:53,422 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,422 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:53,422 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:53,423 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:53,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:53,423 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:53,424 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,424 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:53,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:53,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:53,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:53,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a89a526[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:53,425 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:53,425 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,426 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,426 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:53,426 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:53,426 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:53,426 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,427 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:53,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:53,431 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-21 01:55:53,431 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:53,431 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:53,431 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:53,431 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:53,431 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:53,431 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:53,431 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:53,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:53,432 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:53,432 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:53,432 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:53,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:53,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:53,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:53,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:53,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:53,435 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:53,435 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:53,435 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:53,435 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:53,435 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:53,437 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,437 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:53,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:53,437 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:53,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:53,438 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,440 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,444 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,444 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:53,444 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:53,444 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,444 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:53,444 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:53,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:53,444 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,444 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:53,444 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:53,444 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:53,445 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:53,445 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:53,444 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:53,444 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:53,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:53,454 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/2320955025e84c9baf812c3a6e444600 2023-04-21 01:55:53,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:53,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042213455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:53,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042213456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:53,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042213460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,468 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:53,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042213462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:53,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042213465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,545 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:53,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:53,547 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:53,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:53,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:53,548 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:53,548 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:53,548 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:53,548 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:53,549 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,549 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:53,550 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:53,550 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:53,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:53,550 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,550 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:53,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:53,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:53,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:53,551 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:53,551 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:53,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:53,552 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:53,552 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:53,552 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:53,552 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:53,552 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:53,552 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:53,552 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:53,552 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:53,552 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:53,552 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:53,552 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:53,552 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,552 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:53,552 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:53,555 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:53,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:53,555 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:53,556 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,556 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:53,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:53,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12989ef3[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:53,557 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:53,557 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,558 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,558 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,558 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:53,558 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:53,558 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:53,558 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:53,560 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,560 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:53,560 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:53,560 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:53,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:53,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:53,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:53,562 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:53,562 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:53,562 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:53,562 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:53,562 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:53,564 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,564 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:53,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,564 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:53,564 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:53,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,566 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,569 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:53,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:53,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:53,569 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:53,569 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,569 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:53,569 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:53,569 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:53,569 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:53,569 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:53,569 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,569 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:53,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:53,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:53,569 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:53,570 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:53,569 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,570 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:53,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042213566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:53,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042213566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:53,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042213566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:53,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042213569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:53,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60398 deadline: 1682042213575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,670 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:53,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:53,672 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:53,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:53,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:53,673 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:53,673 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:53,673 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:53,673 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:53,674 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,674 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:53,675 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,675 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:53,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:53,675 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,676 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:53,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:53,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:53,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:53,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:53,677 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:53,677 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:53,677 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:53,677 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:53,679 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,679 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:53,679 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:53,679 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:53,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:53,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3211be5b[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:53,681 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:53,681 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,682 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,682 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:53,682 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:53,682 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:53,682 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,682 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:53,684 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:53,684 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:53,684 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,684 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:53,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:53,686 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:53,686 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:53,686 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:53,687 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:53,687 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:53,688 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,688 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,688 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:53,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:53,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:53,689 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:53,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,691 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,694 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:53,694 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:53,694 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:53,694 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:53,694 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,694 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:53,694 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:53,694 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:53,694 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:53,694 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:53,695 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:53,695 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,695 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:53,695 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:53,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042213775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:53,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60438 deadline: 1682042213776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:53,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60420 deadline: 1682042213777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:53,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60410 deadline: 1682042213780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:53,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042213784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,795 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:53,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:53,797 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:53,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:53,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:53,798 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:53,799 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:53,799 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:53,799 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:53,799 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,799 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:53,800 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:53,800 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:53,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:53,801 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,801 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:53,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:53,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:53,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:53,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:53,804 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:53,804 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:53,804 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:53,804 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:53,804 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:53,804 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:53,804 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:53,804 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:53,804 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:53,805 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:53,805 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:53,805 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,805 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:53,805 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:53,808 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:53,808 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:53,809 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,809 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:53,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:53,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21a3dd83[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:53,810 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:53,810 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,811 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,811 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:53,811 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:53,811 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:53,812 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,812 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:53,813 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:53,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:53,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:53,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:53,817 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:53,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:53,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:53,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:53,817 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:53,819 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,819 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:53,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,819 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:53,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:53,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:53,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:53,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:53,819 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,822 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:53,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:53,834 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:53,834 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:53,834 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:53,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:53,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:53,834 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:53,834 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:53,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:53,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:53,864 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/0ca18c2b1f5a4580a559c07c6e29714e 2023-04-21 01:55:53,935 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:53,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:53,937 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:53,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:53,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:53,938 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:53,938 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:53,938 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:53,938 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:53,938 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,938 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:53,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,939 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:53,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:53,940 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,940 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:53,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:53,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:53,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:53,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:53,941 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:53,941 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:53,941 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:53,942 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:53,942 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:53,943 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:53,943 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:53,943 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:53,943 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:53,943 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:53,943 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:53,943 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,943 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:53,943 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:53,944 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,944 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:53,945 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,945 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:53,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:53,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56f4ec04[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:53,946 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:53,946 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,948 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,948 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:53,948 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,949 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:53,949 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:53,949 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:53,951 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,951 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:53,951 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:53,951 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:53,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:53,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:53,954 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:53,954 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:53,954 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:53,954 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:53,954 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:53,955 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,955 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:53,955 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:53,956 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:53,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,958 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,960 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,960 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:53,960 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:53,960 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:53,961 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:53,960 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:53,960 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:53,961 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:53,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:53,961 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:53,961 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:53,961 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:53,961 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,961 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:53,961 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:53,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:54,061 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:54,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:54,063 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:54,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:54,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:54,064 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:54,064 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:54,064 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:54,064 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:54,065 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,065 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:54,066 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:54,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:54,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,067 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:54,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:54,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:54,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:54,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:54,070 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:54,070 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:54,070 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:54,070 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:54,070 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:54,070 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:54,070 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,070 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:54,070 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:54,072 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,072 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:54,073 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,073 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:54,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:54,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cd9f2f[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:54,074 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:54,075 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,076 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,076 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:54,076 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:54,076 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:54,076 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,076 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:54,078 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:54,078 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:54,078 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,078 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:54,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:54,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:54,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:54,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:54,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:54,080 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:54,080 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:54,080 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:54,080 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:54,080 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:54,082 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,082 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,082 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:54,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:54,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:54,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:54,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:54,082 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:54,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:54,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:54,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042214082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:54,086 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,089 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,089 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:54,089 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:54,089 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,089 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,089 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:54,089 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:54,089 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:54,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:54,089 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:54,089 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:54,089 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,090 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,090 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:54,090 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:54,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:54,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:54,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042214084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,091 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:54,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042214084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:54,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042214086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:54,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042214092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,190 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:54,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:54,192 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:54,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:54,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:54,194 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:54,194 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:54,194 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:54,194 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:54,194 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,194 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:54,195 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:54,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:54,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:54,196 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:54,196 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,196 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:54,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:54,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:54,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:54,198 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:54,198 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:54,198 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:54,198 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:54,199 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:54,199 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:54,199 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:54,199 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:54,199 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:54,199 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:54,199 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:54,199 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:54,199 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,199 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:54,199 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:54,202 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,202 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:54,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:54,202 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,202 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:54,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:54,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:54,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:54,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:54,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f110a0b[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:54,204 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:54,204 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,205 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,205 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:54,205 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:54,205 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:54,205 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,205 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:54,207 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,207 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:54,207 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:54,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:54,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:54,207 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:54,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:54,211 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:54,211 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:54,211 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:54,211 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:54,211 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:54,213 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,213 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:54,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,213 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:54,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:54,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:54,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:54,215 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:54,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,217 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,221 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,221 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:54,221 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:54,221 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,221 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,221 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,221 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:54,221 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:54,222 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:54,222 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:54,222 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:54,222 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:54,222 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:54,222 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:54,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:54,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:54,277 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/cc860b29ee7d45c69988a50a17c8b812 2023-04-21 01:55:54,281 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/2320955025e84c9baf812c3a6e444600 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/2320955025e84c9baf812c3a6e444600 2023-04-21 01:55:54,284 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/2320955025e84c9baf812c3a6e444600, entries=200, sequenceid=281, filesize=14.3 K 2023-04-21 01:55:54,285 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/0ca18c2b1f5a4580a559c07c6e29714e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/0ca18c2b1f5a4580a559c07c6e29714e 2023-04-21 01:55:54,288 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/0ca18c2b1f5a4580a559c07c6e29714e, entries=150, sequenceid=281, filesize=11.9 K 2023-04-21 01:55:54,289 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/cc860b29ee7d45c69988a50a17c8b812 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/cc860b29ee7d45c69988a50a17c8b812 2023-04-21 01:55:54,292 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/cc860b29ee7d45c69988a50a17c8b812, entries=150, sequenceid=281, filesize=11.9 K 2023-04-21 01:55:54,293 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 82ecbe07d2c67bb21d6485daedbb8c82 in 862ms, sequenceid=281, compaction requested=true 2023-04-21 01:55:54,293 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:54,293 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:54,294 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:54,294 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-21 01:55:54,294 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-21 01:55:54,294 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-21 01:55:54,298 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 65788 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-21 01:55:54,298 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/A is initiating minor compaction (all files) 2023-04-21 01:55:54,298 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/A in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:54,298 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/581948ab03c548caa931805755ae847a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/3b76d5f9c6914d319cdc147700c927f2, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/fee29d68ff884884bfc44d7029fbe390, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/127dbbf28cf84e3fb4fc1a8a1eda1ec4, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/2320955025e84c9baf812c3a6e444600] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=64.2 K 2023-04-21 01:55:54,299 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60958 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-21 01:55:54,299 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/B is initiating minor compaction (all files) 2023-04-21 01:55:54,299 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/B in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:54,299 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/deb34bf218ef4281adc0660c94c0e93c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/eba2941809924d38a66a050fa808718f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/c4caf60449bc4610bcc89b20a3bee837, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/31e0bba869a74c609f528163607a31d6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/0ca18c2b1f5a4580a559c07c6e29714e] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=59.5 K 2023-04-21 01:55:54,299 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting deb34bf218ef4281adc0660c94c0e93c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682042150264 2023-04-21 01:55:54,299 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 581948ab03c548caa931805755ae847a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682042150264 2023-04-21 01:55:54,300 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting eba2941809924d38a66a050fa808718f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682042150905 2023-04-21 01:55:54,300 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 3b76d5f9c6914d319cdc147700c927f2, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682042150905 2023-04-21 01:55:54,300 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting c4caf60449bc4610bcc89b20a3bee837, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1682042151029 2023-04-21 01:55:54,300 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting fee29d68ff884884bfc44d7029fbe390, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1682042151029 2023-04-21 01:55:54,300 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 31e0bba869a74c609f528163607a31d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1682042151654 2023-04-21 01:55:54,301 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 127dbbf28cf84e3fb4fc1a8a1eda1ec4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1682042151654 2023-04-21 01:55:54,301 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 2320955025e84c9baf812c3a6e444600, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1682042152806 2023-04-21 01:55:54,301 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 0ca18c2b1f5a4580a559c07c6e29714e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1682042152808 2023-04-21 01:55:54,314 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#A#compaction#470 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:54,314 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#B#compaction#471 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:54,322 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:54,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:54,326 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:54,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:54,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:54,328 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:54,328 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:54,328 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:54,328 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:54,328 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,328 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:54,328 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/2c31e1ad4f3e4d90ad65c7295f7d2cbd as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/2c31e1ad4f3e4d90ad65c7295f7d2cbd 2023-04-21 01:55:54,329 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/afa637f7573b4ff887d2c0ec337115dd as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/afa637f7573b4ff887d2c0ec337115dd 2023-04-21 01:55:54,331 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:54,331 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:54,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:54,332 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,332 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:54,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:54,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:54,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:54,333 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:54,333 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:54,334 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:54,334 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:54,334 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:54,334 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:54,334 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:54,334 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:54,334 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:54,334 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-21 01:55:54,334 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:54,334 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:54,334 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:54,335 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:54,335 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:54,335 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:54,335 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/B of 82ecbe07d2c67bb21d6485daedbb8c82 into 2c31e1ad4f3e4d90ad65c7295f7d2cbd(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:54,335 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:54,335 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/B, priority=11, startTime=1682042154294; duration=0sec 2023-04-21 01:55:54,335 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/A of 82ecbe07d2c67bb21d6485daedbb8c82 into afa637f7573b4ff887d2c0ec337115dd(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:54,335 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:54,335 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:54,335 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-21 01:55:54,335 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/A, priority=11, startTime=1682042154293; duration=0sec 2023-04-21 01:55:54,335 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:54,337 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60924 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-21 01:55:54,337 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/C is initiating minor compaction (all files) 2023-04-21 01:55:54,337 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/C in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:54,337 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/9e7c9b829ee549208408c12a7ee3907a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/543e89f6686c4fbbb587e874c01c764b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/d892e94d92054b8ba87da132d233373c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/72f60ab3a79f41c79e4d743efbc27619, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/cc860b29ee7d45c69988a50a17c8b812] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=59.5 K 2023-04-21 01:55:54,337 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 9e7c9b829ee549208408c12a7ee3907a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682042150264 2023-04-21 01:55:54,338 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 543e89f6686c4fbbb587e874c01c764b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1682042150905 2023-04-21 01:55:54,338 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting d892e94d92054b8ba87da132d233373c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1682042151029 2023-04-21 01:55:54,340 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 72f60ab3a79f41c79e4d743efbc27619, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1682042151654 2023-04-21 01:55:54,340 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting cc860b29ee7d45c69988a50a17c8b812, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1682042152808 2023-04-21 01:55:54,352 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/e3e294b5d11d4eef8319f89e3817d170 2023-04-21 01:55:54,353 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#C#compaction#473 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:54,366 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/1d116c965a424f15bad5304e725597c1 2023-04-21 01:55:54,368 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/0a3d90d3894a47a59c0568d33aa25268 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/0a3d90d3894a47a59c0568d33aa25268 2023-04-21 01:55:54,374 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/C of 82ecbe07d2c67bb21d6485daedbb8c82 into 0a3d90d3894a47a59c0568d33aa25268(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:54,374 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:54,374 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/C, priority=11, startTime=1682042154294; duration=0sec 2023-04-21 01:55:54,375 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:54,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:54,590 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:54,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:54,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042214604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:54,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042214605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:54,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042214606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:54,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60462 deadline: 1682042214606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:54,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042214607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:54,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042214710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:54,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042214711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:54,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042214711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:54,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042214713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:54,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042214714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,784 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/d1a32cfe547241119d6d955f9aeab7e3 2023-04-21 01:55:54,788 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/e3e294b5d11d4eef8319f89e3817d170 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/e3e294b5d11d4eef8319f89e3817d170 2023-04-21 01:55:54,792 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/e3e294b5d11d4eef8319f89e3817d170, entries=150, sequenceid=297, filesize=11.9 K 2023-04-21 01:55:54,793 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/1d116c965a424f15bad5304e725597c1 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/1d116c965a424f15bad5304e725597c1 2023-04-21 01:55:54,796 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/1d116c965a424f15bad5304e725597c1, entries=150, sequenceid=297, filesize=11.9 K 2023-04-21 01:55:54,797 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/d1a32cfe547241119d6d955f9aeab7e3 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/d1a32cfe547241119d6d955f9aeab7e3 2023-04-21 01:55:54,800 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/d1a32cfe547241119d6d955f9aeab7e3, entries=150, sequenceid=297, filesize=11.9 K 2023-04-21 01:55:54,801 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 82ecbe07d2c67bb21d6485daedbb8c82 in 467ms, sequenceid=297, compaction requested=false 2023-04-21 01:55:54,801 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:54,801 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:54,801 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:54,801 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:54,801 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,801 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:54,801 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:54,804 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,804 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,805 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,805 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:54,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:54,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:54,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6197af[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:54,806 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:54,806 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,807 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,808 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:54,808 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:54,808 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:54,808 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,808 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:54,809 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,809 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:54,809 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:54,809 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:54,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:54,812 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:54,812 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:54,812 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:54,812 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:54,812 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:54,815 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,815 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:54,815 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:54,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:54,815 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,816 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,818 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:54,818 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,819 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:54,819 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:54,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:54,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:54,818 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:54,819 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:54,819 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:54,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:54,819 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:54,819 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:54,819 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:54,819 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,819 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,819 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,819 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:54,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:54,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:54,919 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-21 01:55:54,919 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:54,919 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:54,919 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:54,919 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:54,920 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:54,920 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:54,920 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:54,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:54,922 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:54,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:54,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:54,924 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:54,924 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:54,924 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:54,924 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:54,925 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,925 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:54,926 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:54,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:54,926 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:54,926 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,926 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:54,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:54,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:54,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:54,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:54,928 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:54,928 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:54,928 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:54,928 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:54,928 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:54,928 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:54,928 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:54,928 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:54,929 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:54,929 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:54,929 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:54,929 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,929 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:54,929 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:54,930 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,930 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:54,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:54,930 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,930 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:54,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:54,933 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:54,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7184df6b[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:54,933 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,935 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,935 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,935 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:54,935 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:54,935 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:54,935 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:54,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:54,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042214929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,937 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:54,937 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:54,937 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,937 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:54,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:54,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:54,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:54,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:54,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:54,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:54,939 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:54,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:54,940 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:54,940 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:54,940 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:54,941 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,941 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,941 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:54,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,941 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:54,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:54,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:54,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:54,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:54,943 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:54,944 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:54,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042214934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,945 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:54,945 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:54,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042214936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,945 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:54,945 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:54,945 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:54,945 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:54,945 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042214936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,945 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:54,946 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:54,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:54,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:54,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:54,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:54,945 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:54,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042214936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,946 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:54,946 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:54,946 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:54,946 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,946 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:54,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:54,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:54,946 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:55,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042215037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,046 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:55,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:55,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:55,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042215045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,048 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:55,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:55,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:55,049 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:55,049 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:55,049 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:55,050 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:55,050 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,050 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,052 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,052 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:55,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:55,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:55,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:55,052 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:55,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042215047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,052 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,053 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:55,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:55,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042215047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042215047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:55,054 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:55,054 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:55,054 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:55,054 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:55,054 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:55,054 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:55,054 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:55,054 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:55,054 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:55,054 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:55,054 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:55,054 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:55,054 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:55,054 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,054 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:55,055 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:55,056 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:55,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:55,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:55,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d5c1293[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:55,058 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:55,058 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,059 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,059 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:55,059 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:55,060 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:55,060 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,060 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:55,061 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:55,061 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:55,061 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:55,061 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:55,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:55,063 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:55,063 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:55,063 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:55,063 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:55,063 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:55,066 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,066 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:55,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,066 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:55,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:55,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:55,067 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:55,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,068 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,070 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:55,070 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:55,070 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:55,070 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:55,070 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,070 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,070 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,070 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,070 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,070 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:55,070 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:55,070 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:55,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:55,070 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:55,071 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:55,171 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:55,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:55,173 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:55,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:55,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:55,174 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:55,174 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:55,174 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:55,174 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:55,175 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,175 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,178 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:55,178 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:55,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:55,178 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,178 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:55,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:55,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:55,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:55,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:55,180 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:55,180 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:55,180 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:55,180 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:55,180 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:55,180 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:55,180 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:55,180 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:55,180 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:55,180 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:55,180 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:55,180 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,180 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:55,180 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:55,186 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:55,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:55,186 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:55,187 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,187 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:55,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:55,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:55,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:55,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a2a0261[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:55,188 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:55,188 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,189 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:55,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:55,190 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,190 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:55,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:55,191 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:55,191 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:55,191 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:55,191 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:55,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:55,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:55,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:55,193 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:55,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:55,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:55,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:55,193 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:55,195 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,195 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:55,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,195 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:55,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,197 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:55,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,199 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,202 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,202 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,202 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:55,202 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:55,203 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:55,202 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:55,202 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:55,202 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:55,203 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,203 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:55,203 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:55,203 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,203 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,203 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:55,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:55,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042215242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:55,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042215248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:55,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042215254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:55,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042215254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:55,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042215255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,303 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:55,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:55,305 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:55,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:55,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:55,307 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:55,307 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:55,307 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:55,307 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:55,307 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,307 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,308 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:55,309 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:55,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:55,309 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,309 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:55,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:55,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:55,310 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:55,310 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:55,311 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:55,311 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:55,311 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:55,311 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:55,311 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:55,311 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:55,311 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:55,311 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:55,311 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:55,311 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,311 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:55,311 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:55,313 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,313 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:55,313 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,313 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:55,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:55,315 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:55,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@267a3464[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:55,315 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,316 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,316 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:55,316 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:55,316 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:55,316 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,316 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:55,318 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:55,318 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:55,318 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:55,319 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:55,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:55,324 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:55,324 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:55,324 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:55,324 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:55,324 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:55,325 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,325 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:55,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,325 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:55,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,327 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,328 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,329 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/c15afd13e5104b468a00125966a611b0 2023-04-21 01:55:55,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,332 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:55,332 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:55,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:55,333 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,333 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,333 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:55,333 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:55,333 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:55,333 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:55,333 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,333 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,333 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,333 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,333 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:55,334 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:55,341 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/e1f3b65db04342aa9d1b26174dfe9357 2023-04-21 01:55:55,434 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:55,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:55,436 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:55,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:55,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:55,437 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:55,437 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:55,437 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:55,437 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:55,438 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,438 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,439 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,439 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:55,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:55,439 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,439 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:55,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:55,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:55,440 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:55,440 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:55,440 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:55,440 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:55,441 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:55,441 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:55,441 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:55,441 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:55,441 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:55,441 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:55,441 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:55,441 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,441 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:55,441 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:55,443 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,443 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:55,443 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,443 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:55,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:55,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71d2d847[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:55,445 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:55,445 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,446 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,446 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:55,446 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:55,446 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:55,446 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,446 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:55,447 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,447 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:55,447 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:55,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:55,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:55,447 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:55,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:55,450 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:55,450 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:55,450 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:55,450 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:55,450 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:55,452 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,452 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,452 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:55,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:55,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:55,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:55,452 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:55,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,453 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,456 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,456 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:55,456 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:55,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:55,456 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,457 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:55,457 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,457 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,457 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,457 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,457 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,457 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:55,457 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:55,457 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:55,457 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:55,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:55,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:55,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:55,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042215547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,557 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:55,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:55,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:55,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042215553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,559 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:55,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:55,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:55,560 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:55,560 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:55,560 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:55,561 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:55,561 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,561 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,562 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:55,562 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:55,562 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,563 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:55,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:55,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:55,564 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:55,564 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:55,564 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:55,564 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:55,564 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:55,564 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:55,564 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:55,564 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:55,564 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:55,564 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:55,564 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:55,564 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,564 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:55,564 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:55,566 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,566 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:55,566 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:55,566 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:55,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:55,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ce79464[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:55,567 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:55,567 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,572 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,572 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:55,572 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:55,572 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,572 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:55,572 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:55,573 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:55,573 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,573 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:55,573 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:55,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:55,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:55,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:55,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:55,576 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:55,576 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:55,576 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:55,576 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:55,576 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:55,577 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,577 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:55,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,577 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:55,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:55,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042215573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:55,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:55,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042215574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:55,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042215574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:55,579 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,584 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,587 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,587 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:55,587 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,587 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:55,587 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:55,587 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:55,587 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:55,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:55,587 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:55,587 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,587 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,587 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,587 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,587 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:55,587 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:55,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:55,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:55,687 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:55,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:55,690 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:55,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:55,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:55,691 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:55,691 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:55,691 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:55,691 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:55,691 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,691 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,694 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:55,694 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:55,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:55,694 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,694 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:55,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:55,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:55,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:55,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:55,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:55,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:55,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:55,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:55,696 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:55,696 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:55,696 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:55,696 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:55,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:55,697 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:55,697 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,697 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:55,697 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:55,698 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,698 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:55,699 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,699 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:55,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:55,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76a4b693[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:55,700 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:55,700 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,701 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,701 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:55,701 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:55,701 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:55,703 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:55,703 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:55,703 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,703 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:55,703 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:55,703 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:55,703 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:55,703 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:55,703 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:55,704 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:55,704 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:55,707 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,707 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,707 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:55,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:55,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:55,708 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:55,711 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,713 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,713 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:55,713 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,713 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,713 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,713 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,713 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:55,713 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,713 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:55,713 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:55,714 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:55,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:55,714 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:55,714 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:55,714 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:55,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:55,753 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/40c2bc780dc3486babea2a1ca8564817 2023-04-21 01:55:55,757 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/c15afd13e5104b468a00125966a611b0 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/c15afd13e5104b468a00125966a611b0 2023-04-21 01:55:55,761 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/c15afd13e5104b468a00125966a611b0, entries=200, sequenceid=317, filesize=14.3 K 2023-04-21 01:55:55,761 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/e1f3b65db04342aa9d1b26174dfe9357 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/e1f3b65db04342aa9d1b26174dfe9357 2023-04-21 01:55:55,765 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/e1f3b65db04342aa9d1b26174dfe9357, entries=150, sequenceid=317, filesize=11.9 K 2023-04-21 01:55:55,766 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/40c2bc780dc3486babea2a1ca8564817 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/40c2bc780dc3486babea2a1ca8564817 2023-04-21 01:55:55,770 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/40c2bc780dc3486babea2a1ca8564817, entries=150, sequenceid=317, filesize=11.9 K 2023-04-21 01:55:55,771 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 82ecbe07d2c67bb21d6485daedbb8c82 in 852ms, sequenceid=317, compaction requested=true 2023-04-21 01:55:55,771 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:55,771 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:55,771 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:55,771 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:55,771 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:55,772 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:55,772 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39646 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:55,772 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/A is initiating minor compaction (all files) 2023-04-21 01:55:55,772 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/A in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:55,773 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:55,773 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/afa637f7573b4ff887d2c0ec337115dd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/e3e294b5d11d4eef8319f89e3817d170, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/c15afd13e5104b468a00125966a611b0] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=38.7 K 2023-04-21 01:55:55,773 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/B is initiating minor compaction (all files) 2023-04-21 01:55:55,773 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/B in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:55,773 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/2c31e1ad4f3e4d90ad65c7295f7d2cbd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/1d116c965a424f15bad5304e725597c1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/e1f3b65db04342aa9d1b26174dfe9357] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=36.3 K 2023-04-21 01:55:55,773 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting afa637f7573b4ff887d2c0ec337115dd, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1682042152808 2023-04-21 01:55:55,775 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 2c31e1ad4f3e4d90ad65c7295f7d2cbd, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1682042152808 2023-04-21 01:55:55,775 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting e3e294b5d11d4eef8319f89e3817d170, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1682042153455 2023-04-21 01:55:55,776 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 1d116c965a424f15bad5304e725597c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1682042153455 2023-04-21 01:55:55,776 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting c15afd13e5104b468a00125966a611b0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682042154599 2023-04-21 01:55:55,776 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting e1f3b65db04342aa9d1b26174dfe9357, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682042154604 2023-04-21 01:55:55,784 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#B#compaction#479 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:55,784 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#A#compaction#480 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:55,802 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/57ad4d58756e48368ba29f8e0f0ddfc0 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/57ad4d58756e48368ba29f8e0f0ddfc0 2023-04-21 01:55:55,807 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/B of 82ecbe07d2c67bb21d6485daedbb8c82 into 57ad4d58756e48368ba29f8e0f0ddfc0(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:55,807 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:55,807 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/B, priority=13, startTime=1682042155771; duration=0sec 2023-04-21 01:55:55,808 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:55,808 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:55:55,808 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:55:55,809 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/C is initiating minor compaction (all files) 2023-04-21 01:55:55,809 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/C in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:55,809 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/0a3d90d3894a47a59c0568d33aa25268, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/d1a32cfe547241119d6d955f9aeab7e3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/40c2bc780dc3486babea2a1ca8564817] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=36.3 K 2023-04-21 01:55:55,809 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 0a3d90d3894a47a59c0568d33aa25268, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1682042152808 2023-04-21 01:55:55,809 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting d1a32cfe547241119d6d955f9aeab7e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1682042153455 2023-04-21 01:55:55,810 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 40c2bc780dc3486babea2a1ca8564817, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682042154604 2023-04-21 01:55:55,814 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:55,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:55,815 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#C#compaction#481 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:55,816 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:55,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:55,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:55,820 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:55,820 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:55,820 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:55,820 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:55,821 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,821 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,822 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:55,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:55,823 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:55,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:55,824 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:55,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:55,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:55,824 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:55,824 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:55,824 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:55,824 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:55,824 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:55,824 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:55,825 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:55,825 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-21 01:55:55,825 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:55,825 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:55,825 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:55,825 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:55,825 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:55,825 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:55,825 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:55,825 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:55,828 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/278ca9335cb84866b0a59b7f8b6607ed as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/278ca9335cb84866b0a59b7f8b6607ed 2023-04-21 01:55:55,834 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/C of 82ecbe07d2c67bb21d6485daedbb8c82 into 278ca9335cb84866b0a59b7f8b6607ed(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:55,834 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:55,834 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/C, priority=13, startTime=1682042155772; duration=0sec 2023-04-21 01:55:55,834 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:56,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:56,059 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:56,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:56,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:56,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042216089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:56,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042216086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:56,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:56,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042216118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:56,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:56,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042216118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:56,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:56,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042216118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:56,199 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/cc40a6a4e4a54596b2c6cb30b80a4f2c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/cc40a6a4e4a54596b2c6cb30b80a4f2c 2023-04-21 01:55:56,203 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/A of 82ecbe07d2c67bb21d6485daedbb8c82 into cc40a6a4e4a54596b2c6cb30b80a4f2c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:56,203 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:56,203 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/A, priority=13, startTime=1682042155771; duration=0sec 2023-04-21 01:55:56,203 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:56,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:56,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042216220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:56,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:56,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042216220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:56,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:56,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042216225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:56,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:56,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042216225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:56,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:56,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042216225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:56,242 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/85308d14c08b44868e17d9dbff9d3627 2023-04-21 01:55:56,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:56,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042216421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:56,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:56,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042216422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:56,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:56,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042216427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:56,431 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:56,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042216428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:56,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:56,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042216430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:56,653 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/73810eb66eb844dcb0a2e87af62c415f 2023-04-21 01:55:56,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:56,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042216726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:56,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:56,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042216727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:56,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:56,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042216730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:56,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:56,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042216733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:56,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:56,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042216734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,064 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/05df64cf16da4ad0b8512a22665724d2 2023-04-21 01:55:57,068 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/85308d14c08b44868e17d9dbff9d3627 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/85308d14c08b44868e17d9dbff9d3627 2023-04-21 01:55:57,072 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/85308d14c08b44868e17d9dbff9d3627, entries=150, sequenceid=333, filesize=11.9 K 2023-04-21 01:55:57,073 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/73810eb66eb844dcb0a2e87af62c415f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/73810eb66eb844dcb0a2e87af62c415f 2023-04-21 01:55:57,076 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/73810eb66eb844dcb0a2e87af62c415f, entries=150, sequenceid=333, filesize=11.9 K 2023-04-21 01:55:57,077 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/05df64cf16da4ad0b8512a22665724d2 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/05df64cf16da4ad0b8512a22665724d2 2023-04-21 01:55:57,080 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/05df64cf16da4ad0b8512a22665724d2, entries=150, sequenceid=333, filesize=11.9 K 2023-04-21 01:55:57,081 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 82ecbe07d2c67bb21d6485daedbb8c82 in 1256ms, sequenceid=333, compaction requested=false 2023-04-21 01:55:57,081 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:57,081 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:57,081 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:57,081 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:57,081 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,081 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:57,081 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:57,083 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,083 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:57,083 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,083 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:57,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:57,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:57,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:57,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10a6537[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:57,085 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:57,085 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,088 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,088 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:57,088 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:57,088 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:57,088 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,089 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:57,089 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,089 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:57,090 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:57,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:57,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:57,090 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:57,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:57,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:57,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:57,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:57,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:57,092 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:57,092 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:57,092 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:57,092 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:57,092 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:57,093 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,093 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,093 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:57,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:57,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:57,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:57,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:57,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:57,094 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:57,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,095 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:57,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,097 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,097 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,097 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,097 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,097 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:57,098 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:57,098 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:57,098 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:57,097 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:57,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:57,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:57,097 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,098 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,098 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:57,098 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:57,098 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:57,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:57,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:57,198 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:57,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:57,200 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:57,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:57,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:57,201 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:57,202 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:57,202 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:57,202 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:57,202 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,202 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,204 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:57,204 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,205 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,205 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:57,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,205 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:57,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:57,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:57,206 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:57,206 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:57,206 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:57,206 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:57,206 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:57,206 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:57,206 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:57,206 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-21 01:55:57,206 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:57,206 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:57,207 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:57,207 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:57,207 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:57,207 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:57,218 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/137560afffbc40f19438fead7857bc3d 2023-04-21 01:55:57,233 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/1a5eae03743441e998bfc6269d95f592 2023-04-21 01:55:57,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:57,237 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:57,252 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/324503e654214f12ab54583ad766f62f 2023-04-21 01:55:57,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042217251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,257 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/137560afffbc40f19438fead7857bc3d as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/137560afffbc40f19438fead7857bc3d 2023-04-21 01:55:57,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042217254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,261 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/137560afffbc40f19438fead7857bc3d, entries=150, sequenceid=352, filesize=11.9 K 2023-04-21 01:55:57,261 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/1a5eae03743441e998bfc6269d95f592 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/1a5eae03743441e998bfc6269d95f592 2023-04-21 01:55:57,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042217255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,265 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/1a5eae03743441e998bfc6269d95f592, entries=150, sequenceid=352, filesize=11.9 K 2023-04-21 01:55:57,265 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042217256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,265 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042217257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,266 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/324503e654214f12ab54583ad766f62f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/324503e654214f12ab54583ad766f62f 2023-04-21 01:55:57,278 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/324503e654214f12ab54583ad766f62f, entries=150, sequenceid=352, filesize=11.9 K 2023-04-21 01:55:57,279 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 82ecbe07d2c67bb21d6485daedbb8c82 in 73ms, sequenceid=352, compaction requested=true 2023-04-21 01:55:57,279 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:57,279 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:57,279 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:57,279 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:57,279 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,279 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:57,279 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:57,281 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,281 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:57,281 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,281 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:57,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:57,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66fd85b1[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:57,283 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:57,283 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,284 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,284 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:57,284 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,284 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:57,284 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:57,284 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:57,291 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,291 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:57,291 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:57,292 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:57,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:57,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:57,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:57,295 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:57,295 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:57,295 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:57,295 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:57,295 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:57,298 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,298 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:57,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:57,298 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:57,299 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,300 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,302 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:57,302 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:57,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:57,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:57,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:57,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:57,304 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,304 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,304 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,304 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,304 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,304 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,304 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,304 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:57,304 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:57,304 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:57,304 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:57,304 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:57,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:57,364 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-21 01:55:57,365 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:57,365 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:57,365 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:57,365 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:57,365 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:57,365 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:57,372 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=369 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/c180aa476b0d48c7ba289138b88f3af9 2023-04-21 01:55:57,387 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=369 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/e804cfe08c63423ea7b8dd99e4e6d156 2023-04-21 01:55:57,397 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=369 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/ce9ddfe4f6f8494ba48a076918cdfd57 2023-04-21 01:55:57,401 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/c180aa476b0d48c7ba289138b88f3af9 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/c180aa476b0d48c7ba289138b88f3af9 2023-04-21 01:55:57,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042217389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042217390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,404 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/c180aa476b0d48c7ba289138b88f3af9, entries=250, sequenceid=369, filesize=16.7 K 2023-04-21 01:55:57,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042217391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,405 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:57,405 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/e804cfe08c63423ea7b8dd99e4e6d156 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/e804cfe08c63423ea7b8dd99e4e6d156 2023-04-21 01:55:57,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:57,407 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:57,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:57,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:57,407 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:57,408 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:57,408 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:57,408 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:57,408 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,408 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042217397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,411 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:57,411 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:57,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:57,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,412 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,412 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:57,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:57,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:57,413 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/e804cfe08c63423ea7b8dd99e4e6d156, entries=150, sequenceid=369, filesize=11.9 K 2023-04-21 01:55:57,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042217404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,414 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/ce9ddfe4f6f8494ba48a076918cdfd57 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/ce9ddfe4f6f8494ba48a076918cdfd57 2023-04-21 01:55:57,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:57,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:57,416 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:57,416 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:57,416 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:57,416 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,417 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:57,417 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:57,419 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,419 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:57,419 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,419 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:57,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:57,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@560979be[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:57,422 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:57,423 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,424 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,424 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:57,424 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:57,424 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,424 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:57,424 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:57,426 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:57,426 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:57,426 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,426 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:57,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:57,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:57,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:57,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:57,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:57,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:57,429 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:57,429 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:57,429 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:57,429 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:57,429 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:57,429 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/ce9ddfe4f6f8494ba48a076918cdfd57, entries=150, sequenceid=369, filesize=11.9 K 2023-04-21 01:55:57,430 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 82ecbe07d2c67bb21d6485daedbb8c82 in 66ms, sequenceid=369, compaction requested=true 2023-04-21 01:55:57,430 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:57,430 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:57,430 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:57,430 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:55:57,430 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:55:57,430 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-21 01:55:57,431 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,431 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:57,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,431 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:57,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:57,432 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:57,432 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54374 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:55:57,432 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/A is initiating minor compaction (all files) 2023-04-21 01:55:57,432 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:55:57,432 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/B is initiating minor compaction (all files) 2023-04-21 01:55:57,432 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/B in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:57,433 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/57ad4d58756e48368ba29f8e0f0ddfc0, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/73810eb66eb844dcb0a2e87af62c415f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/1a5eae03743441e998bfc6269d95f592, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/e804cfe08c63423ea7b8dd99e4e6d156] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=48.3 K 2023-04-21 01:55:57,432 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/A in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,433 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/cc40a6a4e4a54596b2c6cb30b80a4f2c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/85308d14c08b44868e17d9dbff9d3627, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/137560afffbc40f19438fead7857bc3d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/c180aa476b0d48c7ba289138b88f3af9] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=53.1 K 2023-04-21 01:55:57,433 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 57ad4d58756e48368ba29f8e0f0ddfc0, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682042154604 2023-04-21 01:55:57,433 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting cc40a6a4e4a54596b2c6cb30b80a4f2c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682042154604 2023-04-21 01:55:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:57,434 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,434 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 73810eb66eb844dcb0a2e87af62c415f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1682042154926 2023-04-21 01:55:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,434 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 85308d14c08b44868e17d9dbff9d3627, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1682042154926 2023-04-21 01:55:57,434 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 137560afffbc40f19438fead7857bc3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1682042156085 2023-04-21 01:55:57,434 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 1a5eae03743441e998bfc6269d95f592, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1682042156085 2023-04-21 01:55:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,435 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting c180aa476b0d48c7ba289138b88f3af9, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1682042157244 2023-04-21 01:55:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:57,435 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting e804cfe08c63423ea7b8dd99e4e6d156, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1682042157244 2023-04-21 01:55:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,437 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:57,437 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:57,437 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,437 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,437 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:57,437 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,437 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,437 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,437 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,437 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:57,437 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:57,437 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:57,438 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:57,438 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:57,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:57,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:57,445 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#A#compaction#491 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:57,446 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#B#compaction#492 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:57,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:57,511 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-21 01:55:57,512 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:57,512 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:57,512 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:57,512 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:57,512 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:57,512 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:57,522 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/4a79424b961b45a7b624c8089bde37c7 2023-04-21 01:55:57,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042217526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042217527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,538 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:57,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:57,540 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:57,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:57,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:57,541 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:57,541 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:57,541 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:57,541 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:57,541 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,541 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,543 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:57,543 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:57,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,543 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,543 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042217535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042217536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,544 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:57,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:57,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:57,544 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:57,544 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:57,544 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:57,545 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:57,545 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:57,545 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:57,545 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:57,545 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:57,545 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:57,545 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:57,545 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:57,545 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,545 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:57,545 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:57,546 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,546 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:57,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:57,546 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,546 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:57,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:57,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:57,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:57,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042217537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:57,548 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:57,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ca5dc2[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:57,548 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,549 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,549 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:57,549 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:57,549 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:57,549 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,550 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:57,552 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:57,552 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:57,552 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,552 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:57,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:57,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:57,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:57,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:57,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:57,555 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:57,555 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:57,555 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:57,555 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:57,555 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:57,556 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,556 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:57,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:57,556 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:57,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:57,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:57,557 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:57,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,558 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,560 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,560 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,560 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,560 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,560 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:57,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:57,560 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,561 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:57,561 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:57,561 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,561 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,561 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:57,561 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:57,562 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:57,562 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:57,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:57,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:57,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042217638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042217638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042217644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042217644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042217649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,661 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:57,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:57,663 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:57,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:57,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:57,664 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:57,665 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:57,665 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:57,665 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:57,665 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,665 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,666 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:57,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:57,667 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,667 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:57,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:57,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:57,668 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:57,668 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:57,668 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:57,668 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:57,668 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:57,668 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:57,668 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:57,668 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:57,668 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:57,668 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:57,669 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:57,669 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,669 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:57,669 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:57,671 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,671 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:57,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:57,672 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,672 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:57,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:57,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:57,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:57,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@573ba391[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:57,673 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:57,673 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,675 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,675 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:57,675 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:57,675 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:57,675 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,676 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:57,677 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,677 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:57,677 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:57,677 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:57,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:57,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:57,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:57,679 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:57,680 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:57,680 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:57,680 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:57,680 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:57,681 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,681 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:57,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,682 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,682 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,683 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:57,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:57,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:57,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,688 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:57,688 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,688 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:57,688 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:57,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:57,688 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:57,688 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:57,688 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,688 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,688 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,688 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,689 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,689 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:57,689 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:57,689 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:57,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:57,789 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:57,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:57,791 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:57,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:57,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:57,792 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:57,792 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:57,792 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:57,792 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:57,793 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,793 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,794 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,794 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:57,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:57,795 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,795 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:57,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:57,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:57,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:57,796 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:57,796 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:57,796 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:57,796 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:57,796 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:57,796 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:57,796 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:57,796 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:57,797 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:57,797 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:57,797 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:57,797 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,797 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:57,797 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:57,798 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,798 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:57,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:57,799 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,799 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:57,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:57,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:57,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:57,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:57,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d6164ca[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:57,800 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:57,800 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,801 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,801 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:57,801 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:57,801 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:57,801 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,801 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:57,804 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:57,804 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:57,804 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,804 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:57,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:57,807 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:57,807 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:57,807 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:57,807 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:57,807 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:57,808 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,808 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:57,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,808 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:57,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:57,810 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:57,812 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,816 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,816 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,816 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,816 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:57,816 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:57,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:57,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:57,818 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,818 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,818 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:57,818 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,818 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:57,818 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:57,818 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:57,818 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:57,818 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:57,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042217844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042217844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042217852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042217852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:57,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042217854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,866 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/c3074f25ce2e441380f420b67713f202 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/c3074f25ce2e441380f420b67713f202 2023-04-21 01:55:57,868 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/c9c6b0d5652d4fe99c0cbde8460bcd46 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/c9c6b0d5652d4fe99c0cbde8460bcd46 2023-04-21 01:55:57,872 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/B of 82ecbe07d2c67bb21d6485daedbb8c82 into c3074f25ce2e441380f420b67713f202(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:57,872 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:57,872 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/B, priority=12, startTime=1682042157430; duration=0sec 2023-04-21 01:55:57,872 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:57,872 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:55:57,874 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/A of 82ecbe07d2c67bb21d6485daedbb8c82 into c9c6b0d5652d4fe99c0cbde8460bcd46(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:57,874 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:57,874 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/A, priority=12, startTime=1682042157430; duration=0sec 2023-04-21 01:55:57,874 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:57,875 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:55:57,875 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/C is initiating minor compaction (all files) 2023-04-21 01:55:57,875 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/C in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:57,875 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/278ca9335cb84866b0a59b7f8b6607ed, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/05df64cf16da4ad0b8512a22665724d2, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/324503e654214f12ab54583ad766f62f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/ce9ddfe4f6f8494ba48a076918cdfd57] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=48.3 K 2023-04-21 01:55:57,875 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 278ca9335cb84866b0a59b7f8b6607ed, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682042154604 2023-04-21 01:55:57,876 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 05df64cf16da4ad0b8512a22665724d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1682042154926 2023-04-21 01:55:57,876 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 324503e654214f12ab54583ad766f62f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1682042156085 2023-04-21 01:55:57,876 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting ce9ddfe4f6f8494ba48a076918cdfd57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1682042157244 2023-04-21 01:55:57,884 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#C#compaction#495 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:57,918 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:57,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:57,920 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:57,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:57,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:57,921 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:57,922 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:57,922 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:57,922 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:57,922 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,922 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,923 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,923 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:57,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:57,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:57,923 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,924 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:57,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:57,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:57,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:57,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:57,925 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:57,925 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:57,925 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:57,925 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:57,925 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:57,925 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:57,925 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:57,925 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:57,926 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:57,926 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:57,926 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:57,926 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,926 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:57,926 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:57,928 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,930 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:57,930 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,930 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,931 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/b0e163b2684a4744b20d0e555a90af4a 2023-04-21 01:55:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:57,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:57,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40382251[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:57,933 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:57,933 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,934 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,934 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:57,934 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:57,934 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:57,934 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,934 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:57,935 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:57,935 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:57,935 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:57,936 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:57,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:57,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:57,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:57,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:57,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:57,939 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:57,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:57,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:57,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:57,939 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:57,941 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,941 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:57,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,941 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:57,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:57,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:57,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:57,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:57,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:57,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,943 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,944 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:57,947 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:57,947 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:57,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:57,947 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,947 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,947 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:57,947 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:57,948 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,948 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:57,948 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:57,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:57,948 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:57,948 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:57,948 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:57,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:57,948 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:57,948 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:58,048 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:58,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:58,051 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:58,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:58,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:58,052 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:58,052 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:58,052 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:58,052 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:58,052 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,053 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:58,054 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,054 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:58,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:58,054 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,054 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:58,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:58,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:58,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:58,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:58,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:58,055 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:58,055 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:58,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:58,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:58,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:58,056 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:58,056 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:58,056 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:58,056 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:58,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:58,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:58,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:58,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:58,057 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:58,058 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:58,058 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,058 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:58,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:58,060 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:58,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@174b53d9[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:58,060 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,061 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,061 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:58,061 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:58,061 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:58,061 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,062 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:58,068 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:58,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:58,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:58,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:58,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:58,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:58,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:58,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:58,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:58,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:58,071 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:58,071 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:58,071 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:58,071 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:58,071 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:58,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:58,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:58,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:58,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:58,074 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,075 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,079 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:58,079 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:58,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:58,079 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:58,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:58,080 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,080 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:58,080 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:58,080 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,080 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,080 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,080 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,080 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:58,080 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:58,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:58,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:58,080 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:58,080 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:58,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:58,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042218150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:58,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042218150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:58,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042218156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:58,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042218157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:58,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042218158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,180 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:58,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:58,182 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:58,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:58,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:58,184 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:58,184 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:58,184 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:58,184 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:58,184 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,184 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:58,191 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:58,191 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:58,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:58,191 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:58,191 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:58,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:58,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:58,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:58,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:58,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:58,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:58,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:58,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:58,193 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:58,193 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:58,193 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:58,193 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:58,193 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:58,193 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:58,193 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:58,193 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:58,193 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,193 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:58,193 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:58,194 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,194 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:58,195 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,195 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:58,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:58,196 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:58,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d664d8a[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:58,196 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,197 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,197 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:58,197 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:58,197 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:58,197 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,197 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:58,198 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:58,198 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:58,198 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,198 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:58,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:58,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:58,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:58,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:58,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:58,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:58,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:58,201 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:58,201 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:58,201 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:58,201 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:58,201 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:58,204 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,204 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:58,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:58,204 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:58,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:58,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:58,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:58,204 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,205 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:58,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,207 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:58,207 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:58,207 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:58,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:58,207 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:58,207 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:58,207 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:58,207 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:58,208 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:58,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:58,207 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,208 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,208 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,208 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:58,212 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:58,212 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:58,299 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/5fae6e7fd9f0486bbd14964087e95ba3 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/5fae6e7fd9f0486bbd14964087e95ba3 2023-04-21 01:55:58,302 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/C of 82ecbe07d2c67bb21d6485daedbb8c82 into 5fae6e7fd9f0486bbd14964087e95ba3(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:58,302 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:58,303 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/C, priority=12, startTime=1682042157430; duration=0sec 2023-04-21 01:55:58,303 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:58,312 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:58,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:58,314 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:58,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:58,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:58,315 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:58,315 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:58,315 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:58,315 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:58,316 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,316 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:58,317 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:58,317 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:58,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:58,317 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,317 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:58,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:58,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:58,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:58,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:58,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:58,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:58,318 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:58,318 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:58,318 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:58,318 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:58,318 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:58,319 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:58,319 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:58,319 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:58,319 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:58,319 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:58,319 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:58,319 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,319 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:58,319 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:58,320 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,320 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:58,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:58,320 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,320 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:58,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:58,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:58,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b772dd8[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:58,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,324 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,324 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:58,324 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:58,324 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:58,324 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,324 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:58,325 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:58,326 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:58,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:58,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:58,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:58,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:58,328 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:58,328 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:58,328 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:58,328 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:58,328 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:58,329 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:58,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:58,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:58,330 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,330 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,330 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:58,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:58,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:58,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:58,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,331 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:58,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,334 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:58,334 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:58,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:58,334 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,334 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:58,334 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:58,334 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:58,334 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,334 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,334 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,335 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,334 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:58,334 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:58,335 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:58,335 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:58,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:58,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:58,355 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/173d7585983d461c862033c8c2947efc 2023-04-21 01:55:58,359 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/4a79424b961b45a7b624c8089bde37c7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/4a79424b961b45a7b624c8089bde37c7 2023-04-21 01:55:58,363 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/4a79424b961b45a7b624c8089bde37c7, entries=300, sequenceid=387, filesize=19.0 K 2023-04-21 01:55:58,363 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/b0e163b2684a4744b20d0e555a90af4a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/b0e163b2684a4744b20d0e555a90af4a 2023-04-21 01:55:58,366 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/b0e163b2684a4744b20d0e555a90af4a, entries=150, sequenceid=387, filesize=11.9 K 2023-04-21 01:55:58,368 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/173d7585983d461c862033c8c2947efc as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/173d7585983d461c862033c8c2947efc 2023-04-21 01:55:58,372 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/173d7585983d461c862033c8c2947efc, entries=150, sequenceid=387, filesize=11.9 K 2023-04-21 01:55:58,372 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 82ecbe07d2c67bb21d6485daedbb8c82 in 861ms, sequenceid=387, compaction requested=false 2023-04-21 01:55:58,372 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:58,435 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:58,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:58,437 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:58,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:58,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:58,438 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:58,438 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:58,438 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:58,438 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:58,439 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,439 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:58,441 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:58,441 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:58,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:58,441 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,441 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:58,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:58,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:58,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:58,442 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:58,442 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:58,442 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:58,442 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:58,443 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:58,443 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:58,443 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:58,443 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:58,443 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:58,443 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:58,443 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-21 01:55:58,443 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:58,444 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:58,444 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:58,444 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:58,444 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:58,444 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:55:58,457 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/054881a30f124116b4bd64bf61ee54b7 2023-04-21 01:55:58,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:58,659 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:58,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:58,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042218676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,685 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:58,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042218679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:58,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042218679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:58,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042218682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:58,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042218682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:58,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042218784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:58,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042218786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:58,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042218789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:58,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042218791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:58,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042218791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,874 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/c1aaa07d67e34a3a96fbeddef441f347 2023-04-21 01:55:58,884 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/6e2a6d56caad40b2862f776847f25550 2023-04-21 01:55:58,888 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/054881a30f124116b4bd64bf61ee54b7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/054881a30f124116b4bd64bf61ee54b7 2023-04-21 01:55:58,891 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/054881a30f124116b4bd64bf61ee54b7, entries=150, sequenceid=404, filesize=11.9 K 2023-04-21 01:55:58,892 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/c1aaa07d67e34a3a96fbeddef441f347 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/c1aaa07d67e34a3a96fbeddef441f347 2023-04-21 01:55:58,895 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/c1aaa07d67e34a3a96fbeddef441f347, entries=150, sequenceid=404, filesize=11.9 K 2023-04-21 01:55:58,896 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/6e2a6d56caad40b2862f776847f25550 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/6e2a6d56caad40b2862f776847f25550 2023-04-21 01:55:58,899 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/6e2a6d56caad40b2862f776847f25550, entries=150, sequenceid=404, filesize=11.9 K 2023-04-21 01:55:58,899 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 82ecbe07d2c67bb21d6485daedbb8c82 in 456ms, sequenceid=404, compaction requested=true 2023-04-21 01:55:58,900 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:58,900 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:58,900 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:58,900 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:58,900 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,900 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:58,900 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:58,901 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:58,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:58,901 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,902 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,902 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:58,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:58,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:58,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:58,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a5cb44d[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:58,904 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:58,904 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,905 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,905 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:58,905 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:58,906 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,906 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:58,906 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:58,907 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,907 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:58,907 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:58,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:58,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:58,907 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:58,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:58,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:58,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:58,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:58,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:58,910 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:58,910 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:58,910 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:58,910 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:58,910 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:58,912 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,912 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:58,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,912 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:58,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:58,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:58,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:58,913 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:58,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:58,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,914 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:58,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:58,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,919 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:58,919 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:58,919 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:58,919 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:58,919 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:58,919 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:58,919 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,919 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:58,919 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:58,919 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:58,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:58,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:58,919 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:58,919 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:58,920 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:58,920 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:59,020 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:59,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:59,021 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-21 01:55:59,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:55:59,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1682042158678, max timestamp=1682042159005Num uniques 859; 2023-04-21 01:55:59,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-21 01:55:59,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1682042158678, max timestamp=1682042159005Num uniques 859; 2023-04-21 01:55:59,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-21 01:55:59,022 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:59,022 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:59,022 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:59,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682042158680, max timestamp=1682042159005Num uniques 859; 2023-04-21 01:55:59,022 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:59,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-21 01:55:59,022 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:59,023 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:59,023 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:59,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:59,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:59,024 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:59,024 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:59,024 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:59,024 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:59,024 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,024 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,028 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:59,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:59,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:59,028 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,029 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,029 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:59,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:59,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,030 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:59,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:59,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:59,030 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:59,030 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:59,030 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:59,032 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:59,032 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:59,032 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:59,032 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:59,032 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:59,032 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:59,032 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:59,032 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:59,032 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,032 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:59,032 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:59,035 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,035 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:59,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:59,035 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,035 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:59,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:59,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:59,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@744e344[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:59,037 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:59,037 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,037 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,038 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:59,038 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:59,038 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:59,038 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,038 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:59,041 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,041 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:59,041 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:59,041 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:59,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:59,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:59,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:59,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:59,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:59,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:59,044 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:59,044 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:59,044 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:59,044 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:59,044 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:59,046 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,046 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,046 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:59,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:59,046 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,047 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:59,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,050 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:59,050 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,050 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:59,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:59,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:59,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:59,050 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:59,050 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:59,050 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:59,050 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:59,051 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:59,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:59,050 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,051 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:59,051 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,051 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,051 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,051 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682042159055, max timestamp=1682042159066Num uniques 859; 2023-04-21 01:55:59,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682042159055, max timestamp=1682042159066Num uniques 859; 2023-04-21 01:55:59,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682042159055, max timestamp=1682042159066Num uniques 859; 2023-04-21 01:55:59,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-21 01:55:59,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-21 01:55:59,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-21 01:55:59,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:59,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042219088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:59,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042219089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:59,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042219089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:59,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042219096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:59,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042219098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,151 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:59,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:59,153 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:59,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:59,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:59,154 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:59,154 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:59,154 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:59,154 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:59,154 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,154 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,156 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,156 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:59,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:59,156 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,156 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:59,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,157 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:59,157 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:59,157 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:59,157 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:59,157 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:59,157 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:59,157 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:59,157 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:59,157 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:59,157 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:59,157 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:59,157 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:59,157 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:59,157 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:59,158 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,158 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:59,158 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:59,159 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,159 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:59,159 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:59,159 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:59,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:59,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28e1f43b[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:59,162 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:59,162 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,163 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,163 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:59,163 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:59,163 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:59,163 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,163 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:59,165 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,165 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:59,165 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:59,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:59,166 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:59,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:59,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:59,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:59,168 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:59,168 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:59,168 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:59,168 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:59,168 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:59,170 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,170 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:59,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,170 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:59,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:59,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:59,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:59,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:59,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,171 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,172 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,173 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,174 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:59,174 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:59,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:59,174 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:59,174 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:59,174 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:59,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:59,174 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:59,174 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,174 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,174 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:59,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:59,174 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:59,174 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,174 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,174 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:59,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:59,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042219199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:59,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042219199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:59,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042219199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:59,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042219204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:59,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042219207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,274 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:59,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:59,277 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:59,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:59,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:59,278 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:59,278 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:59,278 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:59,278 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:59,279 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,279 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,280 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:59,280 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:59,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:59,280 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,280 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:59,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:59,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:59,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:59,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:59,281 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:59,282 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:59,282 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:59,282 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:59,282 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:59,284 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,284 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:59,285 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,285 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:59,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:59,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a8a1512[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:59,286 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:59,286 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,287 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,287 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:59,287 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,287 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:59,287 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:59,288 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:59,289 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:59,289 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,289 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:59,289 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:59,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:59,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:59,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:59,292 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:59,292 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:59,292 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:59,292 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:59,292 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:59,293 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,293 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:59,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:59,294 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:59,294 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:59,295 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,299 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,299 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,299 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:59,299 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:59,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:59,299 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:59,299 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:59,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:59,299 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:59,299 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:59,299 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:59,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:59,299 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,300 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:59,300 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,300 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:59,300 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,400 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:59,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:59,402 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:59,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:59,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:59,403 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:59,404 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:59,404 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:59,404 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:59,404 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,404 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,405 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,405 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:59,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:59,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:59,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,406 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,406 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:59,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,407 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:59,407 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:59,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:59,407 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:59,407 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:59,407 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:59,407 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:59,407 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:59,407 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:59,407 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:59,407 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:59,408 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:59,408 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:59,408 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:59,408 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,408 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:59,408 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:59,409 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:59,410 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,410 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:59,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:59,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:59,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:59,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:59,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755b2cd2[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:59,411 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:59,411 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:59,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042219406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:59,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042219406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:59,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042219407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:59,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042219410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,417 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,417 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:59,417 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:59,417 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,417 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:59,417 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:59,418 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:59,418 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:59,419 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:59,419 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:59,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:59,422 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:59,422 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:59,422 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:59,422 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:59,422 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:59,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:59,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042219416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,424 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:59,424 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,424 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:59,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:59,424 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,426 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,428 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,428 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,428 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,428 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,428 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:59,428 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,428 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:59,428 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:59,428 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:59,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:59,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:59,428 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:59,428 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:59,428 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:59,428 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:59,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:59,429 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,432 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/305ab8981f264caa8c072ad43fab5404 2023-04-21 01:55:59,529 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:59,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:59,531 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:59,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:59,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:59,532 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:59,532 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:59,532 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:59,532 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:59,533 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,533 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,535 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:59,535 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:59,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:59,536 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,536 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,536 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:59,537 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:59,537 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:59,537 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:59,537 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:59,537 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:59,537 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,538 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:59,538 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:59,539 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,539 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:59,539 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,539 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:59,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:59,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:59,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:59,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5525affe[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:59,541 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:59,541 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,542 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:59,544 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:59,544 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:59,544 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:59,544 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,544 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:59,544 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:59,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:59,546 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:59,546 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:59,546 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:59,546 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:59,546 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:59,549 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,549 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:59,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,549 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:59,550 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:59,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,551 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:59,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,553 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,553 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,553 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,553 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,553 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:59,553 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:59,553 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:59,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:59,553 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:59,554 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:59,553 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,554 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,554 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:59,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:55:59,554 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:59,554 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:59,654 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:59,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:59,657 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:59,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:59,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:59,658 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:59,658 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:59,658 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:59,658 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:59,658 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,658 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,660 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:59,660 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:59,660 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,660 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:59,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:59,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:59,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:59,662 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:59,662 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:59,662 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:59,662 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:59,662 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:59,662 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:59,662 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:59,662 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:59,662 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:59,662 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:59,662 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:59,662 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,662 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:59,662 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:59,664 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,664 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:59,664 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,664 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:59,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:59,667 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:59,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72c2d94d[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:59,667 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,670 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,670 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,670 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:59,670 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:59,670 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:59,670 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:59,671 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:59,672 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:59,671 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,672 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:59,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:59,674 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:59,675 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:59,675 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:59,675 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:59,675 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:59,676 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,676 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:59,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:59,676 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:59,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:59,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:59,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:59,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:59,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:59,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,680 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,681 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,684 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,684 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,684 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,684 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,684 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:59,684 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:59,684 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:59,684 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:59,684 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:59,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:59,684 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:59,684 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,685 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,685 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:59,685 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:59,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:59,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:59,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:59,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042219714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:59,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042219715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:59,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042219716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:59,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042219717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,733 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:55:59,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042219724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,785 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:59,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:59,787 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:59,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:59,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:59,788 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:59,788 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:59,788 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:59,788 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:59,790 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,790 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,792 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:59,792 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:59,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:59,793 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,793 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:59,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:59,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:59,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:59,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:59,794 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:59,794 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:59,794 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:59,794 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:59,794 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:59,794 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:59,794 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:59,794 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:55:59,794 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:59,794 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:55:59,794 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:55:59,794 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,794 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:55:59,795 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:55:59,796 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,796 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:59,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:59,796 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,796 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:55:59,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:59,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:59,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:59,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:55:59,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@786a7082[Count = 0] remaining members to acquire global barrier 2023-04-21 01:55:59,797 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:55:59,797 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,798 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,798 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:55:59,799 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,799 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:55:59,799 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:55:59,799 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:55:59,803 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,803 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:55:59,803 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:59,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:59,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:59,803 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:55:59,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:59,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:59,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:59,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:55:59,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:55:59,805 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:55:59,805 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:55:59,805 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:55:59,805 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:55:59,805 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:55:59,807 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,807 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:55:59,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:55:59,807 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:59,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:55:59,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:59,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:59,807 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:55:59,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,809 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:55:59,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:55:59,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,812 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,812 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,812 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:59,812 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,812 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:55:59,812 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:55:59,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:59,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:59,812 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:55:59,812 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:55:59,812 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:55:59,812 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:55:59,812 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:55:59,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:55:59,812 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,813 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:55:59,813 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,844 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/f279fce87bc747ac95aa9571977bdbed 2023-04-21 01:55:59,853 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=10.79 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/376c03460e3841ddb65988776564767d 2023-04-21 01:55:59,860 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/305ab8981f264caa8c072ad43fab5404 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/305ab8981f264caa8c072ad43fab5404 2023-04-21 01:55:59,864 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/305ab8981f264caa8c072ad43fab5404, entries=200, sequenceid=426, filesize=14.3 K 2023-04-21 01:55:59,864 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/f279fce87bc747ac95aa9571977bdbed as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/f279fce87bc747ac95aa9571977bdbed 2023-04-21 01:55:59,867 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/f279fce87bc747ac95aa9571977bdbed, entries=150, sequenceid=426, filesize=11.9 K 2023-04-21 01:55:59,868 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/376c03460e3841ddb65988776564767d as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/376c03460e3841ddb65988776564767d 2023-04-21 01:55:59,871 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/376c03460e3841ddb65988776564767d, entries=150, sequenceid=426, filesize=11.9 K 2023-04-21 01:55:59,872 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~42.10 KB/43106, heapSize ~105.51 KB/108040, currentSize=147.60 KB/151140 for 82ecbe07d2c67bb21d6485daedbb8c82 in 851ms, sequenceid=426, compaction requested=true 2023-04-21 01:55:59,872 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:59,872 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:55:59,872 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:55:59,872 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:55:59,872 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:59,872 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:59,874 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 59390 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:55:59,874 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/A is initiating minor compaction (all files) 2023-04-21 01:55:59,874 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/A in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:59,874 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/c9c6b0d5652d4fe99c0cbde8460bcd46, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/4a79424b961b45a7b624c8089bde37c7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/054881a30f124116b4bd64bf61ee54b7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/305ab8981f264caa8c072ad43fab5404] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=58.0 K 2023-04-21 01:55:59,874 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:55:59,874 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/B is initiating minor compaction (all files) 2023-04-21 01:55:59,874 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/B in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:59,874 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/c3074f25ce2e441380f420b67713f202, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/b0e163b2684a4744b20d0e555a90af4a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/c1aaa07d67e34a3a96fbeddef441f347, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/f279fce87bc747ac95aa9571977bdbed] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=48.5 K 2023-04-21 01:55:59,876 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting c9c6b0d5652d4fe99c0cbde8460bcd46, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1682042157244 2023-04-21 01:55:59,876 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting c3074f25ce2e441380f420b67713f202, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1682042157244 2023-04-21 01:55:59,877 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 4a79424b961b45a7b624c8089bde37c7, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1682042157388 2023-04-21 01:55:59,877 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting b0e163b2684a4744b20d0e555a90af4a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1682042157388 2023-04-21 01:55:59,877 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 054881a30f124116b4bd64bf61ee54b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1682042157524 2023-04-21 01:55:59,877 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting c1aaa07d67e34a3a96fbeddef441f347, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1682042157524 2023-04-21 01:55:59,877 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 305ab8981f264caa8c072ad43fab5404, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1682042158680 2023-04-21 01:55:59,878 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting f279fce87bc747ac95aa9571977bdbed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1682042159004 2023-04-21 01:55:59,891 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#A#compaction#503 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:59,897 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#B#compaction#504 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:55:59,907 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/f05be8d0238a49fd9a94d71dfef5bb66 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/f05be8d0238a49fd9a94d71dfef5bb66 2023-04-21 01:55:59,912 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/A of 82ecbe07d2c67bb21d6485daedbb8c82 into f05be8d0238a49fd9a94d71dfef5bb66(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:55:59,913 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:55:59,913 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:55:59,913 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/A, priority=12, startTime=1682042159872; duration=0sec 2023-04-21 01:55:59,913 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:55:59,913 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:55:59,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:55:59,914 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:55:59,914 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/C is initiating minor compaction (all files) 2023-04-21 01:55:59,915 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/C in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:59,915 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/5fae6e7fd9f0486bbd14964087e95ba3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/173d7585983d461c862033c8c2947efc, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/6e2a6d56caad40b2862f776847f25550, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/376c03460e3841ddb65988776564767d] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=48.4 K 2023-04-21 01:55:59,915 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:55:59,915 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 5fae6e7fd9f0486bbd14964087e95ba3, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1682042157244 2023-04-21 01:55:59,916 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 173d7585983d461c862033c8c2947efc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1682042157388 2023-04-21 01:55:59,916 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 6e2a6d56caad40b2862f776847f25550, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1682042157524 2023-04-21 01:55:59,916 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 376c03460e3841ddb65988776564767d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1682042159004 2023-04-21 01:55:59,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:55:59,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:55:59,917 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:55:59,917 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:59,917 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:55:59,917 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:55:59,917 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,917 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,919 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:55:59,919 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:55:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:55:59,919 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:55:59,919 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:55:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:55:59,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:55:59,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:55:59,920 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:55:59,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:55:59,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:55:59,920 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:55:59,920 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:55:59,920 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:55:59,920 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:55:59,920 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:55:59,920 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:55:59,921 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:55:59,921 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-04-21 01:55:59,921 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:55:59,921 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:59,921 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:55:59,921 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:59,921 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:55:59,921 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:55:59,924 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#C#compaction#506 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:00,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:56:00,226 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:56:00,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:00,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042220229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:00,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042220230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:00,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042220235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:00,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042220235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:00,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042220236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,322 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/ec0256b170b14cb18a884ba5a2366567 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/ec0256b170b14cb18a884ba5a2366567 2023-04-21 01:56:00,325 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/B of 82ecbe07d2c67bb21d6485daedbb8c82 into ec0256b170b14cb18a884ba5a2366567(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:00,325 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:56:00,326 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/B, priority=12, startTime=1682042159872; duration=0sec 2023-04-21 01:56:00,326 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:00,341 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/9624be29bc084a4eaf689f1273b2ef25 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/9624be29bc084a4eaf689f1273b2ef25 2023-04-21 01:56:00,345 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/C of 82ecbe07d2c67bb21d6485daedbb8c82 into 9624be29bc084a4eaf689f1273b2ef25(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:00,345 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:56:00,345 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/C, priority=12, startTime=1682042159872; duration=0sec 2023-04-21 01:56:00,345 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:00,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:00,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042220342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:00,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042220343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,347 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/55402640218c437ebdebd90fa9d9ee42 2023-04-21 01:56:00,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:00,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042220343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:00,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042220343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,360 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/19efbb095a234019b306c287af12ac34 2023-04-21 01:56:00,373 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/1815c310964244689bde97bd3fe045c3 2023-04-21 01:56:00,377 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/55402640218c437ebdebd90fa9d9ee42 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/55402640218c437ebdebd90fa9d9ee42 2023-04-21 01:56:00,380 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/55402640218c437ebdebd90fa9d9ee42, entries=150, sequenceid=464, filesize=11.9 K 2023-04-21 01:56:00,381 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/19efbb095a234019b306c287af12ac34 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/19efbb095a234019b306c287af12ac34 2023-04-21 01:56:00,391 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/19efbb095a234019b306c287af12ac34, entries=150, sequenceid=464, filesize=11.9 K 2023-04-21 01:56:00,392 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/1815c310964244689bde97bd3fe045c3 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1815c310964244689bde97bd3fe045c3 2023-04-21 01:56:00,397 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1815c310964244689bde97bd3fe045c3, entries=150, sequenceid=464, filesize=11.9 K 2023-04-21 01:56:00,398 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=33.54 KB/34350 for 82ecbe07d2c67bb21d6485daedbb8c82 in 477ms, sequenceid=464, compaction requested=false 2023-04-21 01:56:00,398 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:56:00,398 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:00,398 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:00,399 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:00,399 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,399 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:00,399 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:00,403 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,403 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:00,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:00,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:00,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:00,404 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:00,404 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:00,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:00,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:00,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@547beee1[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:00,406 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:00,406 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:00,407 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:00,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:00,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:00,407 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:00,407 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:00,407 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:00,408 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:00,408 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:00,409 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:00,409 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,410 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:00,410 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:00,410 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:00,410 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:00,410 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:00,410 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:00,410 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:00,412 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:00,413 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:00,413 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:00,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:00,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:00,414 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,418 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,422 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:00,422 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:00,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:00,422 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:00,422 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,422 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:00,422 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:00,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:00,422 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:00,422 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:00,423 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,423 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:00,423 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:00,423 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:00,423 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:00,423 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:00,523 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:00,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:00,525 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:00,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:00,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:00,526 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:00,526 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:00,526 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:00,526 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:00,527 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:00,527 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:00,528 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:00,528 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:00,529 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,529 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:00,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:00,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:00,531 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:00,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:00,531 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:00,531 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:00,531 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:00,531 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:00,531 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:00,531 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:00,531 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:56:00,532 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-21 01:56:00,532 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:56:00,532 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:00,532 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:56:00,532 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:00,532 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:56:00,532 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:00,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:56:00,550 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:56:00,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682042160562, max timestamp=1682042160573Num uniques 859; 2023-04-21 01:56:00,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-21 01:56:00,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682042160562, max timestamp=1682042160573Num uniques 859; 2023-04-21 01:56:00,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682042160562, max timestamp=1682042160573Num uniques 859; 2023-04-21 01:56:00,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-21 01:56:00,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-21 01:56:00,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:00,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042220637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:00,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042220643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:00,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042220644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:00,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042220645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:00,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042220748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:00,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042220753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:00,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042220753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:00,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042220753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,943 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/147500606dc148e895b303244786cdbd 2023-04-21 01:56:00,953 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/87d1bf5872a14625818315f187e5275f 2023-04-21 01:56:00,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:00,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042220954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:00,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042220956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:00,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042220956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:00,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:00,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042220956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:01,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042221245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:01,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042221262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:01,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042221264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:01,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042221265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:01,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042221265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,368 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/dfdaf4c55c07453599aa939493dfb8b8 2023-04-21 01:56:01,372 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/147500606dc148e895b303244786cdbd as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/147500606dc148e895b303244786cdbd 2023-04-21 01:56:01,375 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/147500606dc148e895b303244786cdbd, entries=100, sequenceid=474, filesize=9.5 K 2023-04-21 01:56:01,376 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/87d1bf5872a14625818315f187e5275f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/87d1bf5872a14625818315f187e5275f 2023-04-21 01:56:01,380 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/87d1bf5872a14625818315f187e5275f, entries=100, sequenceid=474, filesize=9.5 K 2023-04-21 01:56:01,380 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/dfdaf4c55c07453599aa939493dfb8b8 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/dfdaf4c55c07453599aa939493dfb8b8 2023-04-21 01:56:01,383 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/dfdaf4c55c07453599aa939493dfb8b8, entries=100, sequenceid=474, filesize=9.5 K 2023-04-21 01:56:01,384 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for 82ecbe07d2c67bb21d6485daedbb8c82 in 852ms, sequenceid=474, compaction requested=true 2023-04-21 01:56:01,384 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:56:01,384 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:01,385 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:01,385 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,385 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:01,385 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:01,386 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:01,386 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:01,387 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:01,387 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:01,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:01,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cd9f1cf[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:01,388 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:01,388 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:01,389 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:01,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:01,389 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:01,389 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:01,389 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:01,389 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,389 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:01,390 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:01,390 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,390 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:01,390 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:01,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:01,393 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:01,393 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:01,393 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:01,393 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:01,393 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:01,395 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:01,395 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:01,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:01,395 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:01,396 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:01,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:01,397 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:01,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:01,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,399 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:01,399 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:01,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:01,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:01,399 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:01,399 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:01,400 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:01,400 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:01,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:01,400 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:01,400 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:01,400 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:01,400 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:01,400 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:01,400 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:01,400 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:01,400 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:01,500 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:01,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:01,502 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:01,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:01,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:01,504 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:01,504 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:01,504 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:01,504 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:01,504 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:01,504 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:01,505 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:01,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:01,505 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:01,506 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,506 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:01,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:01,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:01,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:01,506 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:01,507 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:01,507 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:01,507 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:01,507 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:01,507 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:01,507 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:01,507 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:01,507 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:01,507 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:56:01,507 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=154.31 KB heapSize=430.69 KB 2023-04-21 01:56:01,507 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:56:01,508 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:01,508 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:56:01,508 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:01,508 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:56:01,509 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:01,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:56:01,770 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:56:01,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:01,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042221778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:01,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042221779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:01,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042221785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:01,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042221785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:01,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042221886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:01,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042221886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:01,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042221892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:01,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042221892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,925 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=511 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/5d286194cf2544a294986bc41a315c0b 2023-04-21 01:56:01,937 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=511 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/59932d32037d4116b9f947a42a0d075c 2023-04-21 01:56:01,946 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=511 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/31a5b7cd22394d73aa94ab4a63f27533 2023-04-21 01:56:01,950 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/5d286194cf2544a294986bc41a315c0b as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/5d286194cf2544a294986bc41a315c0b 2023-04-21 01:56:01,953 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/5d286194cf2544a294986bc41a315c0b, entries=150, sequenceid=511, filesize=11.9 K 2023-04-21 01:56:01,954 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/59932d32037d4116b9f947a42a0d075c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/59932d32037d4116b9f947a42a0d075c 2023-04-21 01:56:01,958 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/59932d32037d4116b9f947a42a0d075c, entries=150, sequenceid=511, filesize=11.9 K 2023-04-21 01:56:01,959 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/31a5b7cd22394d73aa94ab4a63f27533 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/31a5b7cd22394d73aa94ab4a63f27533 2023-04-21 01:56:01,962 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/31a5b7cd22394d73aa94ab4a63f27533, entries=150, sequenceid=511, filesize=11.9 K 2023-04-21 01:56:01,963 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~430.64 KB/440976, currentSize=40.25 KB/41220 for 82ecbe07d2c67bb21d6485daedbb8c82 in 456ms, sequenceid=511, compaction requested=true 2023-04-21 01:56:01,963 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:56:01,963 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:01,963 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:01,963 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:01,963 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,963 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:01,963 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:01,964 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,964 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:01,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:01,965 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:01,965 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:01,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:01,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:01,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:01,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48b54da8[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:01,968 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:01,968 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:01,970 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:01,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:01,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:01,971 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:01,971 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:01,971 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:01,972 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,972 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:01,973 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:01,973 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,973 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:01,973 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:01,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:01,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:01,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:01,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:01,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:01,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:01,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:01,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:01,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:01,976 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:01,976 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:01,976 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:01,976 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:01,976 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:01,978 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:01,978 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:01,978 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:01,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:01,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:01,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:01,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:01,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:01,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:01,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:01,978 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:01,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:01,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:01,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:01,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:01,979 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:01,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:01,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,985 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:01,985 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:01,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:01,985 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:01,985 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:01,985 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:01,985 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:01,985 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:01,985 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:01,985 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:01,986 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:01,986 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:01,986 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:01,986 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:01,986 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:01,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:01,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:02,086 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:02,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:02,088 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:02,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:02,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:02,089 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:02,089 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:02,089 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:02,089 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:02,090 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:02,090 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:02,091 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:02,091 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:02,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:02,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:02,091 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:02,091 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:02,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:02,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:02,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:02,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:02,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:02,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:02,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:02,092 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:02,092 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:02,092 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:02,092 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:02,092 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:02,093 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:02,093 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:56:02,093 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-21 01:56:02,093 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:56:02,093 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:02,093 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:56:02,093 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:02,093 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:56:02,093 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:02,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:56:02,104 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:56:02,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682042162112, max timestamp=1682042162143Num uniques 859; 2023-04-21 01:56:02,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682042162112, max timestamp=1682042162143Num uniques 859; 2023-04-21 01:56:02,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-21 01:56:02,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682042162112, max timestamp=1682042162143Num uniques 859; 2023-04-21 01:56:02,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-21 01:56:02,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-21 01:56:02,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:02,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042222169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:02,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:02,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042222169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:02,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:02,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042222174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:02,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:02,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042222179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:02,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:02,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042222280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:02,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:02,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042222283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:02,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:02,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042222283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:02,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:02,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042222287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:02,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:02,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042222484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:02,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:02,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042222490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:02,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:02,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042222490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:02,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:02,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042222493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:02,503 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/ef62e9bb1ec041378b76776e00a227eb 2023-04-21 01:56:02,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:02,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042222788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:02,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:02,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042222795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:02,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:02,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042222797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:02,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:02,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042222799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:02,916 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/8bd6553ddcc7484cbd21a79140e6ab11 2023-04-21 01:56:03,267 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:03,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60438 deadline: 1682042223262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,269 DEBUG [Thread-3260] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4170 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:56:03,302 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:03,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042223297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,310 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:03,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042223304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:03,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042223306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:03,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042223309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,327 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/1f84d2253d48419cae1a7ccea7636f9f 2023-04-21 01:56:03,332 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/ef62e9bb1ec041378b76776e00a227eb as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/ef62e9bb1ec041378b76776e00a227eb 2023-04-21 01:56:03,335 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/ef62e9bb1ec041378b76776e00a227eb, entries=150, sequenceid=520, filesize=11.9 K 2023-04-21 01:56:03,336 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/8bd6553ddcc7484cbd21a79140e6ab11 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/8bd6553ddcc7484cbd21a79140e6ab11 2023-04-21 01:56:03,340 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/8bd6553ddcc7484cbd21a79140e6ab11, entries=150, sequenceid=520, filesize=11.9 K 2023-04-21 01:56:03,340 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/1f84d2253d48419cae1a7ccea7636f9f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1f84d2253d48419cae1a7ccea7636f9f 2023-04-21 01:56:03,343 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1f84d2253d48419cae1a7ccea7636f9f, entries=150, sequenceid=520, filesize=11.9 K 2023-04-21 01:56:03,344 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for 82ecbe07d2c67bb21d6485daedbb8c82 in 1251ms, sequenceid=520, compaction requested=true 2023-04-21 01:56:03,344 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:56:03,344 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:03,344 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:03,345 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:03,345 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,345 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:03,345 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:03,346 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,346 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:03,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:03,346 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,347 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:03,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:03,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:03,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:03,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:03,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63f0ca5c[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:03,348 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:03,348 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,349 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,349 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:03,349 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:03,349 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:03,349 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,349 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:03,352 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,352 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:03,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:03,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:03,352 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:03,353 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:03,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:03,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:03,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:03,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:03,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:03,356 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:03,356 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:03,356 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:03,356 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:03,356 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:03,358 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,358 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:03,358 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:03,358 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:03,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:03,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,359 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:03,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,362 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,362 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:03,362 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:03,362 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:03,362 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:03,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:03,362 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,362 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,362 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,362 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:03,363 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:03,363 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:03,363 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:03,364 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:03,364 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:03,464 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:03,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:03,466 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:03,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:03,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:03,467 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:03,467 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:03,467 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:03,467 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:03,468 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,468 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:03,469 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:03,469 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:03,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:03,470 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,470 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:03,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:03,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:03,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:03,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:03,471 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:03,471 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:03,471 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:03,471 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:03,471 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:03,471 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:03,472 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:03,472 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:03,472 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:56:03,472 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=147.60 KB heapSize=410.77 KB 2023-04-21 01:56:03,472 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:56:03,472 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:03,472 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:56:03,472 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:03,472 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:56:03,472 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:03,481 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/66fa2c6f932e4795ae3fe7dbb56e20c3 2023-04-21 01:56:03,497 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/3d5adc279e744d9f85b5c7066c960913 2023-04-21 01:56:03,507 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/25e5260573d94e0b9f284a9452665dbe 2023-04-21 01:56:03,513 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/66fa2c6f932e4795ae3fe7dbb56e20c3 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/66fa2c6f932e4795ae3fe7dbb56e20c3 2023-04-21 01:56:03,517 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/66fa2c6f932e4795ae3fe7dbb56e20c3, entries=150, sequenceid=556, filesize=11.9 K 2023-04-21 01:56:03,519 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/3d5adc279e744d9f85b5c7066c960913 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/3d5adc279e744d9f85b5c7066c960913 2023-04-21 01:56:03,522 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/3d5adc279e744d9f85b5c7066c960913, entries=150, sequenceid=556, filesize=11.9 K 2023-04-21 01:56:03,523 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/25e5260573d94e0b9f284a9452665dbe as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/25e5260573d94e0b9f284a9452665dbe 2023-04-21 01:56:03,526 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/25e5260573d94e0b9f284a9452665dbe, entries=150, sequenceid=556, filesize=11.9 K 2023-04-21 01:56:03,526 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=0 B/0 for 82ecbe07d2c67bb21d6485daedbb8c82 in 54ms, sequenceid=556, compaction requested=true 2023-04-21 01:56:03,526 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:56:03,526 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:03,527 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:03,527 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:03,527 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,527 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:03,527 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:03,530 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:03,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:03,530 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,530 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,530 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:03,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:03,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:03,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:03,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:03,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68f73c25[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:03,534 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:03,534 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,535 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,536 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:03,536 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:03,536 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:03,536 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,536 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:03,537 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,537 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:03,537 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:03,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:03,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:03,537 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:03,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:03,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:03,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:03,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:03,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:03,541 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:03,542 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:03,542 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:03,542 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:03,542 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:03,543 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,543 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:03,543 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:03,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:03,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:03,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:03,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:03,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,545 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,546 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,549 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:03,549 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:03,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:03,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:03,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:03,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:03,550 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,550 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:03,550 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:03,550 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,550 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:03,550 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:03,550 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:03,550 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,550 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,550 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,550 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:03,550 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:03,651 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:03,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:03,653 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:03,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:03,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:03,654 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:03,654 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:03,654 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:03,654 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:03,655 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,655 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:03,656 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,656 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:03,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:03,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:03,656 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,656 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:03,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:03,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:03,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:03,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:03,657 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:03,657 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:03,657 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:03,657 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:03,657 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:03,657 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:03,658 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:03,658 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:03,658 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:56:03,658 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:56:03,658 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:03,658 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:03,658 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:03,658 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,658 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:03,658 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:03,659 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,659 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:03,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:03,660 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,660 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:03,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:03,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:03,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:03,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:03,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b43ff7d[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:03,662 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:03,662 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,663 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,663 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:03,663 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:03,663 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,663 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:03,663 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:03,665 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:03,665 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:03,665 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:03,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:03,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:03,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:03,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:03,668 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:03,668 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:03,668 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:03,668 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:03,668 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:03,670 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:03,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:03,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:03,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:03,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,671 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,671 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:03,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,671 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:03,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,672 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:03,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:03,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,677 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:03,677 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:03,677 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:03,677 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:03,677 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:03,678 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:03,678 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:03,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:03,678 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,679 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:03,679 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:03,679 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,679 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,679 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,679 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:03,778 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:03,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:03,780 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:03,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:03,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:03,781 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:03,781 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:03,781 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:03,781 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:03,782 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,782 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:03,784 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,786 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:03,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:03,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:03,786 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,786 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:03,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:03,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:03,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:03,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:03,787 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:03,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:03,788 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:03,788 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:56:03,788 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:56:03,788 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:03,790 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:03,790 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:03,791 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,791 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:03,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:03,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69792bb0[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:03,792 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:03,792 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,793 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,793 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:03,793 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:03,794 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:03,794 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,794 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:03,795 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:03,795 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:03,795 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:03,795 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:03,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:03,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:03,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:03,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:03,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:03,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:03,798 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:03,798 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:03,798 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:03,798 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:03,798 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:03,807 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,807 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:03,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:03,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,808 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:03,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:03,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:03,810 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:03,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,811 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:03,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,813 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,813 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:03,813 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:03,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:03,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:03,813 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:03,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:03,813 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:03,814 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:03,813 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:03,814 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:03,814 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,814 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,814 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,814 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:03,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:03,814 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:03,814 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:03,914 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:03,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:03,916 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:03,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:03,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:03,918 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:03,918 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:03,918 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:03,918 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:03,918 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,918 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:03,919 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,919 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:03,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:03,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:03,919 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,919 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:03,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:03,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:03,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:03,920 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:03,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:03,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:03,921 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:03,921 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:03,921 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:03,921 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:03,921 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:03,921 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:03,921 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:56:03,921 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:56:03,921 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:03,921 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:03,921 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:03,921 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,921 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:03,921 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:03,923 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,923 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:03,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:03,923 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,923 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:03,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:03,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:03,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:03,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:03,925 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:03,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cbebee9[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:03,925 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,933 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,933 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:03,933 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,933 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:03,933 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:03,933 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:03,934 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,934 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:03,934 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:03,935 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:03,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:03,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:03,937 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:03,937 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:03,937 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:03,937 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:03,937 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:03,938 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,938 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:03,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:03,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,938 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:03,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,940 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:03,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:03,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,942 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,942 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:03,942 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:03,942 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:03,942 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:03,942 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:03,942 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:03,942 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:03,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:03,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:03,942 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:03,942 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:03,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:03,942 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,943 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:03,943 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:03,942 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:03,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:03,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:04,043 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:04,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:04,045 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:04,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:04,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:04,046 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:04,046 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:04,046 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:04,046 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:04,047 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,047 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,054 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:04,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:04,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:04,054 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,054 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,055 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:04,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:04,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:04,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:04,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:04,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:04,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:04,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:04,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:04,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:04,056 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:04,056 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:56:04,056 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:56:04,056 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:04,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:04,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:04,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:04,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:04,058 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,058 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,058 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,058 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:04,060 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:04,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5df9ad4f[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:04,060 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,061 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,061 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:04,061 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,061 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:04,061 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:04,062 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:04,063 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,063 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:04,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,063 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:04,063 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:04,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:04,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:04,065 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:04,065 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:04,065 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:04,065 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:04,066 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:04,068 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,068 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,068 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:04,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:04,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:04,068 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,069 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,072 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:04,072 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:04,072 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:04,072 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,072 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:04,072 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:04,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:04,072 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:04,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:04,072 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,072 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:04,072 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,072 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,072 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:04,072 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,172 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:04,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:04,175 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:04,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:04,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:04,176 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:04,176 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:04,176 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:04,176 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:04,176 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,176 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,180 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:04,180 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:04,180 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,180 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:04,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:04,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:04,181 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:04,181 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:04,181 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:04,182 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:04,182 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:04,182 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:04,182 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:56:04,182 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:56:04,182 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:04,182 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:04,182 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:04,182 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,182 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:04,182 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:04,185 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,185 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,185 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,185 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:04,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4955199a[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:04,187 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:04,187 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,188 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,188 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:04,188 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:04,188 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:04,190 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,190 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:04,191 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,191 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,191 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:04,191 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:04,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:04,193 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:04,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:04,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:04,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:04,193 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:04,195 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,195 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,195 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:04,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:04,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:04,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,196 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,198 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,201 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:04,201 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:04,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:04,201 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:04,201 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:04,201 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:04,201 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:04,201 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:04,202 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:04,202 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,202 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:04,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:04,202 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,202 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,202 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,202 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,302 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:04,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:04,304 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:04,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:04,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:04,305 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:04,305 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:04,305 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:04,305 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:04,306 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,306 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,307 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:04,307 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:04,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:04,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,308 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,308 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:04,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:04,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:04,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:04,309 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:04,309 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:04,309 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:04,309 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:04,309 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:04,309 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:04,309 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:56:04,309 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:56:04,309 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:04,309 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:04,309 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:04,310 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,310 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:04,310 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:04,311 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,311 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,311 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,312 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:04,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:04,314 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:04,314 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69518fab[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:04,315 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,315 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,315 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:04,315 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:04,315 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:04,315 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:04,318 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,318 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:04,318 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,318 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:04,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:04,321 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:04,321 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:04,321 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:04,321 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:04,321 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:04,322 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,322 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:04,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,322 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:04,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:04,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,324 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,325 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,327 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,327 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,327 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,327 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:04,327 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:04,327 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:04,327 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,327 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:04,327 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:04,327 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:04,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:04,328 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,328 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,328 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:04,328 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:04,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:04,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:04,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:56:04,333 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-21 01:56:04,334 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:56:04,334 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:04,334 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:56:04,334 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:04,334 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:56:04,334 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:04,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:04,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042224397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042164379, max timestamp=1682042164417Num uniques 859; 2023-04-21 01:56:04,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:04,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042164379, max timestamp=1682042164410Num uniques 859; 2023-04-21 01:56:04,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042164379, max timestamp=1682042164410Num uniques 859; 2023-04-21 01:56:04,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:04,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:04,428 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:04,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:04,430 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:04,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:04,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:04,431 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:04,431 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:04,432 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:04,432 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:04,432 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,432 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,433 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,434 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:04,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:04,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:04,434 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,434 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:04,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:04,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:04,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:04,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:04,435 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:04,435 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:04,435 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:04,435 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:04,435 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:04,435 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:04,436 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:56:04,436 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:56:04,436 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:04,436 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:04,436 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:04,436 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,436 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:04,436 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:04,438 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,438 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,439 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,439 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:04,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:04,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10612d9e[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:04,441 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:04,441 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,442 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,442 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:04,443 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:04,442 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,443 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:04,443 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:04,444 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:04,444 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:04,444 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,444 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:04,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:04,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:04,448 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:04,448 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:04,448 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:04,448 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:04,448 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:04,449 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,449 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,450 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:04,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:04,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:04,450 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,451 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,455 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:04,456 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:04,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:04,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,455 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:04,456 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:04,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:04,456 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:04,456 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:04,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:04,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:04,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:04,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:04,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042224487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:04,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042224497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:04,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042224498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:04,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042224528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,556 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:04,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:04,558 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:04,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:04,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:04,559 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:04,559 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:04,559 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:04,559 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:04,560 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,560 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,561 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:04,561 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:04,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:04,561 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,561 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:04,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:04,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:04,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:04,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:04,562 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:04,562 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:04,562 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:04,562 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:04,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:04,563 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:04,563 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:56:04,563 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:56:04,563 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:04,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:04,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:04,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:04,563 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:04,572 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,572 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,572 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,572 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:04,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:04,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e27908[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:04,573 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:04,574 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,577 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,577 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:04,577 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:04,577 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,577 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:04,577 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:04,578 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,578 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:04,579 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:04,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,579 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:04,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:04,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:04,581 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:04,581 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:04,581 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:04,581 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:04,581 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:04,583 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,583 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,583 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:04,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,583 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:04,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:04,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,584 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,586 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,586 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,586 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,586 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:04,587 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:04,587 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:04,587 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,587 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,587 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,587 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,587 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:04,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:04,587 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:04,587 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:04,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:04,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:04,587 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:04,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:04,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:04,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:04,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042224602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:04,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042224604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,621 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:04,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042224613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,687 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:04,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:04,689 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:04,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:04,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:04,691 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:04,691 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:04,691 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:04,691 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:04,691 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,691 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,693 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,693 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:04,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:04,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:04,694 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,694 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:04,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:04,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:04,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:04,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:04,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:04,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:04,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:04,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:04,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:04,695 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:04,696 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:56:04,696 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:56:04,696 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:04,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:04,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:04,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:04,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:04,698 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,698 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,698 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,698 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:04,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:04,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@611a3947[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:04,700 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:04,700 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,701 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,701 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:04,701 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:04,701 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:04,701 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,701 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:04,703 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,703 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:04,703 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:04,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,703 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:04,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:04,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:04,705 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:04,705 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:04,705 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:04,705 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:04,705 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:04,708 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,708 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,708 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:04,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,709 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:04,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:04,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,711 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,713 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,713 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,713 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,713 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:04,713 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:04,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:04,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:04,713 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,713 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,713 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,713 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:04,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:04,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:04,715 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:04,715 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:04,715 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:04,715 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:04,715 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:04,740 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/a42b3c3115dc44cd80b51fdadedfd338 2023-04-21 01:56:04,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:04,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042224735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:04,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042224807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,815 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:04,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:04,817 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:04,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:04,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:04,818 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:04,819 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:04,819 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:04,819 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:04,819 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,819 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,820 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,820 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:04,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:04,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:04,821 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,821 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:04,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:04,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:04,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:04,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:04,822 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:04,822 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:04,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:04,822 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:04,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042224814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,822 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:04,822 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:04,823 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:04,823 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:56:04,823 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:56:04,823 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:04,823 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:04,823 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:04,823 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,823 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:04,823 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:04,824 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,825 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,825 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,825 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:04,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:04,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@453de18[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:04,826 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:04,826 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,828 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:04,828 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,828 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:04,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:04,829 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:04,830 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,830 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:04,830 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:04,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,830 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:04,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:04,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:04,833 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:04,833 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:04,833 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:04,833 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:04,833 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:04,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:04,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042224824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:04,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:04,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:04,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,836 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,837 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:04,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:04,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:04,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:04,839 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:04,839 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:04,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:04,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:04,839 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:04,840 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:04,840 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,840 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:04,840 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,840 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,840 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,940 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:04,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:04,942 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:04,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:04,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:04,943 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:04,943 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:04,943 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:04,944 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:04,944 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,944 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,946 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,946 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:04,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:04,947 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,947 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:04,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:04,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:04,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:04,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:04,948 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:04,948 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:04,948 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:04,948 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:04,948 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:04,948 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:04,948 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:56:04,948 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:56:04,948 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:04,949 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:04,949 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:04,949 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,949 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:04,949 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:04,950 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,950 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,950 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,950 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:04,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:04,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dc71ed1[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:04,952 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:04,952 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,953 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,954 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:04,954 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:04,954 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:04,954 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,954 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:04,959 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,959 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:04,959 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,959 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:04,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:04,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:04,962 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:04,962 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:04,962 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:04,962 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:04,962 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:04,963 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,963 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:04,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:04,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,963 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:04,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:04,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:04,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:04,964 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:04,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,965 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:04,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:04,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,973 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:04,973 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:04,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:04,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:04,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:04,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:04,975 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,975 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,975 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:04,975 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:04,975 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:04,975 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:04,975 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,975 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:04,975 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:04,975 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:04,975 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:04,975 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:05,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:05,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042225045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:05,075 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:05,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:05,078 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:05,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:05,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:05,079 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:05,079 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:05,079 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:05,079 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:05,079 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:05,079 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:05,082 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:05,082 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:05,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:05,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:05,083 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:05,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:05,083 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:05,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:05,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:05,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:05,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:05,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:05,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:05,084 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:05,084 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:05,084 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:05,084 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:05,084 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:05,084 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:05,084 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. started... 2023-04-21 01:56:05,084 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. as already flushing 2023-04-21 01:56:05,084 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:05,084 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:05,085 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:05,085 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:05,085 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:05,085 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:05,086 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:05,086 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:05,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:05,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:05,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:05,086 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:05,086 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:05,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:05,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:05,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:05,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:05,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:05,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:05,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e7e0e58[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:05,088 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:05,088 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:05,089 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:05,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:05,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:05,089 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:05,089 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:05,089 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:05,089 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:05,089 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:05,091 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:05,091 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:05,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:05,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:05,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:05,091 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:05,091 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:05,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:05,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:05,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:05,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:05,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:05,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:05,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:05,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:05,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:05,095 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:05,095 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:05,095 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:05,095 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:05,095 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:05,098 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:05,098 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:05,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:05,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:05,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:05,098 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:05,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:05,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:05,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:05,098 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:05,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:05,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:05,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:05,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:05,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:05,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:05,100 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:05,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:05,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:05,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:05,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:05,102 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:05,102 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:05,102 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:05,102 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:05,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:05,102 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:05,102 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:05,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:05,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:05,102 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:05,103 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:05,103 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:05,103 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:05,103 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:05,103 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:05,103 DEBUG [Thread-3270] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:05,103 DEBUG [Thread-3270] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:05,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:05,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:05,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:05,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60462 deadline: 1682042225113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:05,123 DEBUG [Thread-3271] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x65816b6e to 127.0.0.1:58383 2023-04-21 01:56:05,123 DEBUG [Thread-3271] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:05,124 DEBUG [Thread-3275] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x720d29f5 to 127.0.0.1:58383 2023-04-21 01:56:05,124 DEBUG [Thread-3275] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:05,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:05,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60410 deadline: 1682042225124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:05,128 DEBUG [Thread-3277] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3011deb3 to 127.0.0.1:58383 2023-04-21 01:56:05,128 DEBUG [Thread-3277] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:05,133 DEBUG [Thread-3273] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x653bb8fc to 127.0.0.1:58383 2023-04-21 01:56:05,133 DEBUG [Thread-3273] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:05,135 DEBUG [Thread-3279] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6e8824d7 to 127.0.0.1:58383 2023-04-21 01:56:05,135 DEBUG [Thread-3279] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:05,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:05,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60398 deadline: 1682042225136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:05,162 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/f54b27f0116d4b579cb5a86e926ca155 2023-04-21 01:56:05,203 DEBUG [Thread-3270] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:05,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:05,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:05,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60420 deadline: 1682042225556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:05,571 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/b1d35bc16c974f35ac69f2ed0a9425c8 2023-04-21 01:56:05,574 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/a42b3c3115dc44cd80b51fdadedfd338 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/a42b3c3115dc44cd80b51fdadedfd338 2023-04-21 01:56:05,577 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/a42b3c3115dc44cd80b51fdadedfd338, entries=350, sequenceid=572, filesize=21.4 K 2023-04-21 01:56:05,578 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/f54b27f0116d4b579cb5a86e926ca155 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/f54b27f0116d4b579cb5a86e926ca155 2023-04-21 01:56:05,581 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/f54b27f0116d4b579cb5a86e926ca155, entries=150, sequenceid=572, filesize=11.9 K 2023-04-21 01:56:05,581 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/b1d35bc16c974f35ac69f2ed0a9425c8 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/b1d35bc16c974f35ac69f2ed0a9425c8 2023-04-21 01:56:05,584 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/b1d35bc16c974f35ac69f2ed0a9425c8, entries=150, sequenceid=572, filesize=11.9 K 2023-04-21 01:56:05,585 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 82ecbe07d2c67bb21d6485daedbb8c82 in 1252ms, sequenceid=572, compaction requested=true 2023-04-21 01:56:05,585 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:56:05,585 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:05,585 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:05,585 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-21 01:56:05,585 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:05,585 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-21 01:56:05,587 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 93644 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-21 01:56:05,587 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83880 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-21 01:56:05,587 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/A is initiating minor compaction (all files) 2023-04-21 01:56:05,587 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/B is initiating minor compaction (all files) 2023-04-21 01:56:05,587 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/A in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:05,587 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/B in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:05,587 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/f05be8d0238a49fd9a94d71dfef5bb66, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/55402640218c437ebdebd90fa9d9ee42, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/147500606dc148e895b303244786cdbd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/5d286194cf2544a294986bc41a315c0b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/ef62e9bb1ec041378b76776e00a227eb, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/66fa2c6f932e4795ae3fe7dbb56e20c3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/a42b3c3115dc44cd80b51fdadedfd338] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=91.4 K 2023-04-21 01:56:05,587 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/ec0256b170b14cb18a884ba5a2366567, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/19efbb095a234019b306c287af12ac34, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/87d1bf5872a14625818315f187e5275f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/59932d32037d4116b9f947a42a0d075c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/8bd6553ddcc7484cbd21a79140e6ab11, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/3d5adc279e744d9f85b5c7066c960913, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/f54b27f0116d4b579cb5a86e926ca155] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=81.9 K 2023-04-21 01:56:05,587 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting f05be8d0238a49fd9a94d71dfef5bb66, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1682042159004 2023-04-21 01:56:05,587 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting ec0256b170b14cb18a884ba5a2366567, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1682042159004 2023-04-21 01:56:05,587 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 55402640218c437ebdebd90fa9d9ee42, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1682042159086 2023-04-21 01:56:05,587 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 19efbb095a234019b306c287af12ac34, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1682042159086 2023-04-21 01:56:05,588 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 147500606dc148e895b303244786cdbd, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682042160234 2023-04-21 01:56:05,588 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 87d1bf5872a14625818315f187e5275f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682042160234 2023-04-21 01:56:05,588 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 5d286194cf2544a294986bc41a315c0b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1682042160635 2023-04-21 01:56:05,588 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 59932d32037d4116b9f947a42a0d075c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1682042160635 2023-04-21 01:56:05,588 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting ef62e9bb1ec041378b76776e00a227eb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1682042161778 2023-04-21 01:56:05,588 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 8bd6553ddcc7484cbd21a79140e6ab11, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1682042161778 2023-04-21 01:56:05,588 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 66fa2c6f932e4795ae3fe7dbb56e20c3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1682042162168 2023-04-21 01:56:05,589 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 3d5adc279e744d9f85b5c7066c960913, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1682042162168 2023-04-21 01:56:05,589 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting a42b3c3115dc44cd80b51fdadedfd338, keycount=350, bloomtype=ROW, size=21.4 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1682042164317 2023-04-21 01:56:05,589 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting f54b27f0116d4b579cb5a86e926ca155, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1682042164322 2023-04-21 01:56:05,598 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#B#compaction#525 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:05,598 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#A#compaction#524 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:05,605 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/0a215ec942df49f9b90f5d77b95b4e14 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/0a215ec942df49f9b90f5d77b95b4e14 2023-04-21 01:56:05,605 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/4ef33ee3d8e541528832dc422f9cf31d as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/4ef33ee3d8e541528832dc422f9cf31d 2023-04-21 01:56:05,609 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/A of 82ecbe07d2c67bb21d6485daedbb8c82 into 0a215ec942df49f9b90f5d77b95b4e14(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:05,609 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/B of 82ecbe07d2c67bb21d6485daedbb8c82 into 4ef33ee3d8e541528832dc422f9cf31d(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:05,609 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:56:05,609 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:56:05,609 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/A, priority=9, startTime=1682042165585; duration=0sec 2023-04-21 01:56:05,609 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/B, priority=9, startTime=1682042165585; duration=0sec 2023-04-21 01:56:05,609 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:05,609 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:05,609 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-21 01:56:05,610 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83846 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-21 01:56:05,611 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 82ecbe07d2c67bb21d6485daedbb8c82/C is initiating minor compaction (all files) 2023-04-21 01:56:05,611 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82ecbe07d2c67bb21d6485daedbb8c82/C in TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:05,611 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/9624be29bc084a4eaf689f1273b2ef25, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1815c310964244689bde97bd3fe045c3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/dfdaf4c55c07453599aa939493dfb8b8, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/31a5b7cd22394d73aa94ab4a63f27533, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1f84d2253d48419cae1a7ccea7636f9f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/25e5260573d94e0b9f284a9452665dbe, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/b1d35bc16c974f35ac69f2ed0a9425c8] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp, totalSize=81.9 K 2023-04-21 01:56:05,611 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 9624be29bc084a4eaf689f1273b2ef25, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1682042159004 2023-04-21 01:56:05,611 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 1815c310964244689bde97bd3fe045c3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1682042159086 2023-04-21 01:56:05,611 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting dfdaf4c55c07453599aa939493dfb8b8, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1682042160234 2023-04-21 01:56:05,612 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 31a5b7cd22394d73aa94ab4a63f27533, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1682042160635 2023-04-21 01:56:05,612 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 1f84d2253d48419cae1a7ccea7636f9f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1682042161778 2023-04-21 01:56:05,612 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 25e5260573d94e0b9f284a9452665dbe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1682042162168 2023-04-21 01:56:05,612 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting b1d35bc16c974f35ac69f2ed0a9425c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1682042164322 2023-04-21 01:56:05,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:56:05,620 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-21 01:56:05,621 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:56:05,621 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-21 01:56:05,621 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:56:05,621 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-21 01:56:05,621 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:56:05,621 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-21 01:56:05,622 DEBUG [Thread-3264] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x461f5f3f to 127.0.0.1:58383 2023-04-21 01:56:05,622 DEBUG [Thread-3264] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:05,625 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82ecbe07d2c67bb21d6485daedbb8c82#C#compaction#526 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:05,631 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=612 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/8a6cf818d10a48fd925a9f482a8d1cf9 2023-04-21 01:56:05,633 DEBUG [Thread-3268] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b475d8d to 127.0.0.1:58383 2023-04-21 01:56:05,633 DEBUG [Thread-3268] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:05,639 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=612 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/910b936b4e684fd9b528d2946ac7c44b 2023-04-21 01:56:05,642 DEBUG [Thread-3262] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0fb168ce to 127.0.0.1:58383 2023-04-21 01:56:05,642 DEBUG [Thread-3262] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:05,649 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=612 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/7fefe5b96f004de4a9bbc87ae3dff0e7 2023-04-21 01:56:05,652 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/8a6cf818d10a48fd925a9f482a8d1cf9 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/8a6cf818d10a48fd925a9f482a8d1cf9 2023-04-21 01:56:05,655 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/8a6cf818d10a48fd925a9f482a8d1cf9, entries=150, sequenceid=612, filesize=11.9 K 2023-04-21 01:56:05,655 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/910b936b4e684fd9b528d2946ac7c44b as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/910b936b4e684fd9b528d2946ac7c44b 2023-04-21 01:56:05,658 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/910b936b4e684fd9b528d2946ac7c44b, entries=150, sequenceid=612, filesize=11.9 K 2023-04-21 01:56:05,659 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/7fefe5b96f004de4a9bbc87ae3dff0e7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/7fefe5b96f004de4a9bbc87ae3dff0e7 2023-04-21 01:56:05,662 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/7fefe5b96f004de4a9bbc87ae3dff0e7, entries=150, sequenceid=612, filesize=11.9 K 2023-04-21 01:56:05,662 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=13.42 KB/13740 for 82ecbe07d2c67bb21d6485daedbb8c82 in 42ms, sequenceid=612, compaction requested=false 2023-04-21 01:56:05,662 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:56:06,035 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/7269e9c3862942db8a726cfc6ddac223 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/7269e9c3862942db8a726cfc6ddac223 2023-04-21 01:56:06,038 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 82ecbe07d2c67bb21d6485daedbb8c82/C of 82ecbe07d2c67bb21d6485daedbb8c82 into 7269e9c3862942db8a726cfc6ddac223(size=13.1 K), total size for store is 25.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:06,038 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:56:06,038 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82., storeName=82ecbe07d2c67bb21d6485daedbb8c82/C, priority=9, startTime=1682042165585; duration=0sec 2023-04-21 01:56:06,039 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:06,560 DEBUG [Thread-3266] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63093e21 to 127.0.0.1:58383 2023-04-21 01:56:06,560 DEBUG [Thread-3266] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:07,302 DEBUG [Thread-3260] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06edd017 to 127.0.0.1:58383 2023-04-21 01:56:07,302 DEBUG [Thread-3260] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:07,302 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-21 01:56:07,302 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 76 2023-04-21 01:56:07,302 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 108 2023-04-21 01:56:07,302 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 102 2023-04-21 01:56:07,302 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 100 2023-04-21 01:56:07,302 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 108 2023-04-21 01:56:07,302 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-21 01:56:07,302 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-21 01:56:07,302 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(402): scanned 1818 2023-04-21 01:56:07,302 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(403): verified 5445 rows 2023-04-21 01:56:07,302 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(402): scanned 1799 2023-04-21 01:56:07,302 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(403): verified 5391 rows 2023-04-21 01:56:07,302 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(402): scanned 1805 2023-04-21 01:56:07,303 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(403): verified 5415 rows 2023-04-21 01:56:07,303 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(402): scanned 1801 2023-04-21 01:56:07,303 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(403): verified 5400 rows 2023-04-21 01:56:07,303 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(402): scanned 1808 2023-04-21 01:56:07,303 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(403): verified 5418 rows 2023-04-21 01:56:07,303 INFO [Listener at localhost/39939] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-21 01:56:07,303 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1c29757d to 127.0.0.1:58383 2023-04-21 01:56:07,303 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:07,307 INFO [Listener at localhost/39939] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-21 01:56:07,307 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-21 01:56:07,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-21 01:56:07,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-21 01:56:07,311 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042167311"}]},"ts":"1682042167311"} 2023-04-21 01:56:07,312 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-21 01:56:07,408 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-21 01:56:07,409 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=82ecbe07d2c67bb21d6485daedbb8c82, UNASSIGN}] 2023-04-21 01:56:07,410 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=82ecbe07d2c67bb21d6485daedbb8c82, UNASSIGN 2023-04-21 01:56:07,411 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=82ecbe07d2c67bb21d6485daedbb8c82, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:07,411 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042167411"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682042167411"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682042167411"}]},"ts":"1682042167411"} 2023-04-21 01:56:07,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-21 01:56:07,412 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399}] 2023-04-21 01:56:07,564 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:56:07,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 82ecbe07d2c67bb21d6485daedbb8c82, disabling compactions & flushes 2023-04-21 01:56:07,564 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:07,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:07,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. after waiting 0 ms 2023-04-21 01:56:07,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:07,564 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 82ecbe07d2c67bb21d6485daedbb8c82 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-21 01:56:07,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=A 2023-04-21 01:56:07,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:07,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=B 2023-04-21 01:56:07,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:07,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82ecbe07d2c67bb21d6485daedbb8c82, store=C 2023-04-21 01:56:07,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:07,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-21 01:56:07,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-21 01:56:07,971 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=620 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/f57cf812a4314d19a284d27aecef6d6b 2023-04-21 01:56:08,380 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=620 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/cd9e41d452d94be3ace178d18722b85e 2023-04-21 01:56:08,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-21 01:56:08,788 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=620 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/484593d709d4492098c8938db62879a3 2023-04-21 01:56:08,791 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/A/f57cf812a4314d19a284d27aecef6d6b as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/f57cf812a4314d19a284d27aecef6d6b 2023-04-21 01:56:08,794 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/f57cf812a4314d19a284d27aecef6d6b, entries=150, sequenceid=620, filesize=11.9 K 2023-04-21 01:56:08,795 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/B/cd9e41d452d94be3ace178d18722b85e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/cd9e41d452d94be3ace178d18722b85e 2023-04-21 01:56:08,798 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/cd9e41d452d94be3ace178d18722b85e, entries=150, sequenceid=620, filesize=11.9 K 2023-04-21 01:56:08,799 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/.tmp/C/484593d709d4492098c8938db62879a3 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/484593d709d4492098c8938db62879a3 2023-04-21 01:56:08,802 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/484593d709d4492098c8938db62879a3, entries=150, sequenceid=620, filesize=11.9 K 2023-04-21 01:56:08,802 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 82ecbe07d2c67bb21d6485daedbb8c82 in 1238ms, sequenceid=620, compaction requested=true 2023-04-21 01:56:08,804 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/0c7e3c7287fd45f58d3510b3713ff792, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/25c0a161746540f0bc6474b64ef5e060, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/fbf94c7344cb49e583bd4436e2030160, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/09b09166d5ca41afaa29427354ac5934, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/46f5cc40e3e84727bf45d2beeac7ed27, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/fef85d2f8db048d9a301c553524d9b27, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/cdfe0fbde7d946fca5fdbeb9907451ca, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/f21c0a3e4b2e45149c8a28200631c3ee, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/a1097efe5aea4696805c09254aeddd4b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/174f50c0245e466685c59a0e3136196b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/97cdfc17065f466f962dd59609ad2897, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/3070dd2fdaf44bc39fc20e81fd9b7d1e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/a60803fc98fb4717959010559c9f748d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/2d6caea96b0c413bb739ce4e07a4db6b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/581948ab03c548caa931805755ae847a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/3b76d5f9c6914d319cdc147700c927f2, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/fee29d68ff884884bfc44d7029fbe390, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/127dbbf28cf84e3fb4fc1a8a1eda1ec4, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/2320955025e84c9baf812c3a6e444600, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/afa637f7573b4ff887d2c0ec337115dd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/e3e294b5d11d4eef8319f89e3817d170, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/c15afd13e5104b468a00125966a611b0, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/cc40a6a4e4a54596b2c6cb30b80a4f2c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/85308d14c08b44868e17d9dbff9d3627, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/137560afffbc40f19438fead7857bc3d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/c180aa476b0d48c7ba289138b88f3af9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/c9c6b0d5652d4fe99c0cbde8460bcd46, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/4a79424b961b45a7b624c8089bde37c7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/054881a30f124116b4bd64bf61ee54b7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/305ab8981f264caa8c072ad43fab5404, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/f05be8d0238a49fd9a94d71dfef5bb66, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/55402640218c437ebdebd90fa9d9ee42, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/147500606dc148e895b303244786cdbd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/5d286194cf2544a294986bc41a315c0b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/ef62e9bb1ec041378b76776e00a227eb, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/66fa2c6f932e4795ae3fe7dbb56e20c3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/a42b3c3115dc44cd80b51fdadedfd338] to archive 2023-04-21 01:56:08,804 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-21 01:56:08,806 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/0c7e3c7287fd45f58d3510b3713ff792 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/0c7e3c7287fd45f58d3510b3713ff792 2023-04-21 01:56:08,807 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/25c0a161746540f0bc6474b64ef5e060 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/25c0a161746540f0bc6474b64ef5e060 2023-04-21 01:56:08,808 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/fbf94c7344cb49e583bd4436e2030160 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/fbf94c7344cb49e583bd4436e2030160 2023-04-21 01:56:08,809 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/09b09166d5ca41afaa29427354ac5934 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/09b09166d5ca41afaa29427354ac5934 2023-04-21 01:56:08,810 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/46f5cc40e3e84727bf45d2beeac7ed27 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/46f5cc40e3e84727bf45d2beeac7ed27 2023-04-21 01:56:08,810 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/fef85d2f8db048d9a301c553524d9b27 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/fef85d2f8db048d9a301c553524d9b27 2023-04-21 01:56:08,811 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/cdfe0fbde7d946fca5fdbeb9907451ca to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/cdfe0fbde7d946fca5fdbeb9907451ca 2023-04-21 01:56:08,812 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/f21c0a3e4b2e45149c8a28200631c3ee to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/f21c0a3e4b2e45149c8a28200631c3ee 2023-04-21 01:56:08,813 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/a1097efe5aea4696805c09254aeddd4b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/a1097efe5aea4696805c09254aeddd4b 2023-04-21 01:56:08,814 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/174f50c0245e466685c59a0e3136196b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/174f50c0245e466685c59a0e3136196b 2023-04-21 01:56:08,815 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/97cdfc17065f466f962dd59609ad2897 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/97cdfc17065f466f962dd59609ad2897 2023-04-21 01:56:08,816 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/3070dd2fdaf44bc39fc20e81fd9b7d1e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/3070dd2fdaf44bc39fc20e81fd9b7d1e 2023-04-21 01:56:08,817 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/a60803fc98fb4717959010559c9f748d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/a60803fc98fb4717959010559c9f748d 2023-04-21 01:56:08,817 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/2d6caea96b0c413bb739ce4e07a4db6b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/2d6caea96b0c413bb739ce4e07a4db6b 2023-04-21 01:56:08,818 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/581948ab03c548caa931805755ae847a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/581948ab03c548caa931805755ae847a 2023-04-21 01:56:08,819 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/3b76d5f9c6914d319cdc147700c927f2 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/3b76d5f9c6914d319cdc147700c927f2 2023-04-21 01:56:08,820 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/fee29d68ff884884bfc44d7029fbe390 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/fee29d68ff884884bfc44d7029fbe390 2023-04-21 01:56:08,821 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/127dbbf28cf84e3fb4fc1a8a1eda1ec4 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/127dbbf28cf84e3fb4fc1a8a1eda1ec4 2023-04-21 01:56:08,822 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/2320955025e84c9baf812c3a6e444600 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/2320955025e84c9baf812c3a6e444600 2023-04-21 01:56:08,822 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/afa637f7573b4ff887d2c0ec337115dd to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/afa637f7573b4ff887d2c0ec337115dd 2023-04-21 01:56:08,823 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/e3e294b5d11d4eef8319f89e3817d170 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/e3e294b5d11d4eef8319f89e3817d170 2023-04-21 01:56:08,824 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/c15afd13e5104b468a00125966a611b0 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/c15afd13e5104b468a00125966a611b0 2023-04-21 01:56:08,825 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/cc40a6a4e4a54596b2c6cb30b80a4f2c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/cc40a6a4e4a54596b2c6cb30b80a4f2c 2023-04-21 01:56:08,826 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/85308d14c08b44868e17d9dbff9d3627 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/85308d14c08b44868e17d9dbff9d3627 2023-04-21 01:56:08,826 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/137560afffbc40f19438fead7857bc3d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/137560afffbc40f19438fead7857bc3d 2023-04-21 01:56:08,827 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/c180aa476b0d48c7ba289138b88f3af9 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/c180aa476b0d48c7ba289138b88f3af9 2023-04-21 01:56:08,828 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/c9c6b0d5652d4fe99c0cbde8460bcd46 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/c9c6b0d5652d4fe99c0cbde8460bcd46 2023-04-21 01:56:08,829 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/4a79424b961b45a7b624c8089bde37c7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/4a79424b961b45a7b624c8089bde37c7 2023-04-21 01:56:08,829 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/054881a30f124116b4bd64bf61ee54b7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/054881a30f124116b4bd64bf61ee54b7 2023-04-21 01:56:08,830 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/305ab8981f264caa8c072ad43fab5404 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/305ab8981f264caa8c072ad43fab5404 2023-04-21 01:56:08,831 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/f05be8d0238a49fd9a94d71dfef5bb66 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/f05be8d0238a49fd9a94d71dfef5bb66 2023-04-21 01:56:08,832 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/55402640218c437ebdebd90fa9d9ee42 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/55402640218c437ebdebd90fa9d9ee42 2023-04-21 01:56:08,833 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/147500606dc148e895b303244786cdbd to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/147500606dc148e895b303244786cdbd 2023-04-21 01:56:08,834 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/5d286194cf2544a294986bc41a315c0b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/5d286194cf2544a294986bc41a315c0b 2023-04-21 01:56:08,834 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/ef62e9bb1ec041378b76776e00a227eb to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/ef62e9bb1ec041378b76776e00a227eb 2023-04-21 01:56:08,835 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/66fa2c6f932e4795ae3fe7dbb56e20c3 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/66fa2c6f932e4795ae3fe7dbb56e20c3 2023-04-21 01:56:08,836 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/a42b3c3115dc44cd80b51fdadedfd338 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/a42b3c3115dc44cd80b51fdadedfd338 2023-04-21 01:56:08,839 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/885a1c3c1cf34c01bd6bad99d90ce34a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/55f5a950fec74790be7d8a2da0f0f1f1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/e729994638cd45a495e2b0c7774a8ad0, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/1586527042ae4a69a7a71ea681654140, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/485d02a215bc41489f4e33ab7194fd68, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/6eb929e77fc34e16b30ac3ceb855c009, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/cd0e647d3c6640b19b8e0bb3833646f8, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/a61ac1d250e34a84a3994364f46dd43b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/b4df412cb07046da9b1b883ac5e580da, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/0f938e9503a642429449ff8a7d80901b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/2813bb7c354f4daab02555fb4ad06472, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/a9e42a8f89374f448ccd47a580434f9a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/23faa52d32f3400a9c85203487eb40d7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/deb34bf218ef4281adc0660c94c0e93c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/5e0bcad35ed44556b0230f352d3d6860, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/eba2941809924d38a66a050fa808718f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/c4caf60449bc4610bcc89b20a3bee837, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/31e0bba869a74c609f528163607a31d6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/2c31e1ad4f3e4d90ad65c7295f7d2cbd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/0ca18c2b1f5a4580a559c07c6e29714e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/1d116c965a424f15bad5304e725597c1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/57ad4d58756e48368ba29f8e0f0ddfc0, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/e1f3b65db04342aa9d1b26174dfe9357, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/73810eb66eb844dcb0a2e87af62c415f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/1a5eae03743441e998bfc6269d95f592, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/c3074f25ce2e441380f420b67713f202, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/e804cfe08c63423ea7b8dd99e4e6d156, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/b0e163b2684a4744b20d0e555a90af4a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/c1aaa07d67e34a3a96fbeddef441f347, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/ec0256b170b14cb18a884ba5a2366567, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/f279fce87bc747ac95aa9571977bdbed, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/19efbb095a234019b306c287af12ac34, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/87d1bf5872a14625818315f187e5275f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/59932d32037d4116b9f947a42a0d075c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/8bd6553ddcc7484cbd21a79140e6ab11, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/3d5adc279e744d9f85b5c7066c960913, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/f54b27f0116d4b579cb5a86e926ca155] to archive 2023-04-21 01:56:08,840 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-21 01:56:08,841 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/885a1c3c1cf34c01bd6bad99d90ce34a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/885a1c3c1cf34c01bd6bad99d90ce34a 2023-04-21 01:56:08,842 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/55f5a950fec74790be7d8a2da0f0f1f1 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/55f5a950fec74790be7d8a2da0f0f1f1 2023-04-21 01:56:08,843 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/e729994638cd45a495e2b0c7774a8ad0 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/e729994638cd45a495e2b0c7774a8ad0 2023-04-21 01:56:08,843 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/1586527042ae4a69a7a71ea681654140 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/1586527042ae4a69a7a71ea681654140 2023-04-21 01:56:08,844 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/485d02a215bc41489f4e33ab7194fd68 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/485d02a215bc41489f4e33ab7194fd68 2023-04-21 01:56:08,845 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/6eb929e77fc34e16b30ac3ceb855c009 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/6eb929e77fc34e16b30ac3ceb855c009 2023-04-21 01:56:08,846 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/cd0e647d3c6640b19b8e0bb3833646f8 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/cd0e647d3c6640b19b8e0bb3833646f8 2023-04-21 01:56:08,847 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/a61ac1d250e34a84a3994364f46dd43b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/a61ac1d250e34a84a3994364f46dd43b 2023-04-21 01:56:08,848 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/b4df412cb07046da9b1b883ac5e580da to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/b4df412cb07046da9b1b883ac5e580da 2023-04-21 01:56:08,849 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/0f938e9503a642429449ff8a7d80901b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/0f938e9503a642429449ff8a7d80901b 2023-04-21 01:56:08,849 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/2813bb7c354f4daab02555fb4ad06472 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/2813bb7c354f4daab02555fb4ad06472 2023-04-21 01:56:08,850 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/a9e42a8f89374f448ccd47a580434f9a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/a9e42a8f89374f448ccd47a580434f9a 2023-04-21 01:56:08,851 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/23faa52d32f3400a9c85203487eb40d7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/23faa52d32f3400a9c85203487eb40d7 2023-04-21 01:56:08,852 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/deb34bf218ef4281adc0660c94c0e93c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/deb34bf218ef4281adc0660c94c0e93c 2023-04-21 01:56:08,853 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/5e0bcad35ed44556b0230f352d3d6860 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/5e0bcad35ed44556b0230f352d3d6860 2023-04-21 01:56:08,854 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/eba2941809924d38a66a050fa808718f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/eba2941809924d38a66a050fa808718f 2023-04-21 01:56:08,855 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/c4caf60449bc4610bcc89b20a3bee837 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/c4caf60449bc4610bcc89b20a3bee837 2023-04-21 01:56:08,856 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/31e0bba869a74c609f528163607a31d6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/31e0bba869a74c609f528163607a31d6 2023-04-21 01:56:08,857 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/2c31e1ad4f3e4d90ad65c7295f7d2cbd to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/2c31e1ad4f3e4d90ad65c7295f7d2cbd 2023-04-21 01:56:08,858 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/0ca18c2b1f5a4580a559c07c6e29714e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/0ca18c2b1f5a4580a559c07c6e29714e 2023-04-21 01:56:08,859 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/1d116c965a424f15bad5304e725597c1 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/1d116c965a424f15bad5304e725597c1 2023-04-21 01:56:08,860 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/57ad4d58756e48368ba29f8e0f0ddfc0 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/57ad4d58756e48368ba29f8e0f0ddfc0 2023-04-21 01:56:08,861 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/e1f3b65db04342aa9d1b26174dfe9357 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/e1f3b65db04342aa9d1b26174dfe9357 2023-04-21 01:56:08,862 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/73810eb66eb844dcb0a2e87af62c415f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/73810eb66eb844dcb0a2e87af62c415f 2023-04-21 01:56:08,863 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/1a5eae03743441e998bfc6269d95f592 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/1a5eae03743441e998bfc6269d95f592 2023-04-21 01:56:08,863 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/c3074f25ce2e441380f420b67713f202 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/c3074f25ce2e441380f420b67713f202 2023-04-21 01:56:08,864 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/e804cfe08c63423ea7b8dd99e4e6d156 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/e804cfe08c63423ea7b8dd99e4e6d156 2023-04-21 01:56:08,865 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/b0e163b2684a4744b20d0e555a90af4a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/b0e163b2684a4744b20d0e555a90af4a 2023-04-21 01:56:08,867 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/c1aaa07d67e34a3a96fbeddef441f347 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/c1aaa07d67e34a3a96fbeddef441f347 2023-04-21 01:56:08,868 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/ec0256b170b14cb18a884ba5a2366567 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/ec0256b170b14cb18a884ba5a2366567 2023-04-21 01:56:08,869 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/f279fce87bc747ac95aa9571977bdbed to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/f279fce87bc747ac95aa9571977bdbed 2023-04-21 01:56:08,869 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/19efbb095a234019b306c287af12ac34 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/19efbb095a234019b306c287af12ac34 2023-04-21 01:56:08,870 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/87d1bf5872a14625818315f187e5275f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/87d1bf5872a14625818315f187e5275f 2023-04-21 01:56:08,871 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/59932d32037d4116b9f947a42a0d075c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/59932d32037d4116b9f947a42a0d075c 2023-04-21 01:56:08,872 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/8bd6553ddcc7484cbd21a79140e6ab11 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/8bd6553ddcc7484cbd21a79140e6ab11 2023-04-21 01:56:08,872 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/3d5adc279e744d9f85b5c7066c960913 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/3d5adc279e744d9f85b5c7066c960913 2023-04-21 01:56:08,873 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/f54b27f0116d4b579cb5a86e926ca155 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/f54b27f0116d4b579cb5a86e926ca155 2023-04-21 01:56:08,876 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/4acfb3cc437f4afb8dddd95c05908554, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/b3ddb399295740d89551fc99e1cd618a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1e7ce63ba7ba4bda83b09afe66644e48, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/01d328a167d6477db09ad7c116881e8f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/f2d9c80ac15e4cc9ad777d0e57f4451e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/efa5050cb5d147709445b7d833fa3ca6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/de556dc581424f3db593444a99d15e24, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/75a91c2fd54142849a5495a7568c0677, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1aca0bc1f6484bccb0c4ce911e4d2a8e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1619b26e46d442e9a43de6e1cb5a1044, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/c5019b55068b4d39919aa065f3e0e3ba, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/2696471ddbb64323a4764baa3fa63200, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/9e7c9b829ee549208408c12a7ee3907a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/8bc105bf3fbb40858bde5153e6092b28, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/543e89f6686c4fbbb587e874c01c764b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/d892e94d92054b8ba87da132d233373c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/72f60ab3a79f41c79e4d743efbc27619, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/0a3d90d3894a47a59c0568d33aa25268, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/cc860b29ee7d45c69988a50a17c8b812, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/d1a32cfe547241119d6d955f9aeab7e3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/278ca9335cb84866b0a59b7f8b6607ed, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/40c2bc780dc3486babea2a1ca8564817, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/05df64cf16da4ad0b8512a22665724d2, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/324503e654214f12ab54583ad766f62f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/5fae6e7fd9f0486bbd14964087e95ba3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/ce9ddfe4f6f8494ba48a076918cdfd57, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/173d7585983d461c862033c8c2947efc, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/6e2a6d56caad40b2862f776847f25550, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/9624be29bc084a4eaf689f1273b2ef25, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/376c03460e3841ddb65988776564767d, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1815c310964244689bde97bd3fe045c3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/dfdaf4c55c07453599aa939493dfb8b8, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/31a5b7cd22394d73aa94ab4a63f27533, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1f84d2253d48419cae1a7ccea7636f9f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/25e5260573d94e0b9f284a9452665dbe, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/b1d35bc16c974f35ac69f2ed0a9425c8] to archive 2023-04-21 01:56:08,877 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-21 01:56:08,878 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/4acfb3cc437f4afb8dddd95c05908554 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/4acfb3cc437f4afb8dddd95c05908554 2023-04-21 01:56:08,879 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/b3ddb399295740d89551fc99e1cd618a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/b3ddb399295740d89551fc99e1cd618a 2023-04-21 01:56:08,879 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1e7ce63ba7ba4bda83b09afe66644e48 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1e7ce63ba7ba4bda83b09afe66644e48 2023-04-21 01:56:08,880 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/01d328a167d6477db09ad7c116881e8f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/01d328a167d6477db09ad7c116881e8f 2023-04-21 01:56:08,881 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/f2d9c80ac15e4cc9ad777d0e57f4451e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/f2d9c80ac15e4cc9ad777d0e57f4451e 2023-04-21 01:56:08,882 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/efa5050cb5d147709445b7d833fa3ca6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/efa5050cb5d147709445b7d833fa3ca6 2023-04-21 01:56:08,883 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/de556dc581424f3db593444a99d15e24 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/de556dc581424f3db593444a99d15e24 2023-04-21 01:56:08,883 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/75a91c2fd54142849a5495a7568c0677 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/75a91c2fd54142849a5495a7568c0677 2023-04-21 01:56:08,884 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1aca0bc1f6484bccb0c4ce911e4d2a8e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1aca0bc1f6484bccb0c4ce911e4d2a8e 2023-04-21 01:56:08,885 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1619b26e46d442e9a43de6e1cb5a1044 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1619b26e46d442e9a43de6e1cb5a1044 2023-04-21 01:56:08,886 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/c5019b55068b4d39919aa065f3e0e3ba to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/c5019b55068b4d39919aa065f3e0e3ba 2023-04-21 01:56:08,886 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/2696471ddbb64323a4764baa3fa63200 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/2696471ddbb64323a4764baa3fa63200 2023-04-21 01:56:08,887 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/9e7c9b829ee549208408c12a7ee3907a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/9e7c9b829ee549208408c12a7ee3907a 2023-04-21 01:56:08,888 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/8bc105bf3fbb40858bde5153e6092b28 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/8bc105bf3fbb40858bde5153e6092b28 2023-04-21 01:56:08,889 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/543e89f6686c4fbbb587e874c01c764b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/543e89f6686c4fbbb587e874c01c764b 2023-04-21 01:56:08,889 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/d892e94d92054b8ba87da132d233373c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/d892e94d92054b8ba87da132d233373c 2023-04-21 01:56:08,890 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/72f60ab3a79f41c79e4d743efbc27619 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/72f60ab3a79f41c79e4d743efbc27619 2023-04-21 01:56:08,891 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/0a3d90d3894a47a59c0568d33aa25268 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/0a3d90d3894a47a59c0568d33aa25268 2023-04-21 01:56:08,892 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/cc860b29ee7d45c69988a50a17c8b812 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/cc860b29ee7d45c69988a50a17c8b812 2023-04-21 01:56:08,892 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/d1a32cfe547241119d6d955f9aeab7e3 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/d1a32cfe547241119d6d955f9aeab7e3 2023-04-21 01:56:08,893 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/278ca9335cb84866b0a59b7f8b6607ed to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/278ca9335cb84866b0a59b7f8b6607ed 2023-04-21 01:56:08,894 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/40c2bc780dc3486babea2a1ca8564817 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/40c2bc780dc3486babea2a1ca8564817 2023-04-21 01:56:08,895 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/05df64cf16da4ad0b8512a22665724d2 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/05df64cf16da4ad0b8512a22665724d2 2023-04-21 01:56:08,895 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/324503e654214f12ab54583ad766f62f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/324503e654214f12ab54583ad766f62f 2023-04-21 01:56:08,896 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/5fae6e7fd9f0486bbd14964087e95ba3 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/5fae6e7fd9f0486bbd14964087e95ba3 2023-04-21 01:56:08,897 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/ce9ddfe4f6f8494ba48a076918cdfd57 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/ce9ddfe4f6f8494ba48a076918cdfd57 2023-04-21 01:56:08,898 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/173d7585983d461c862033c8c2947efc to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/173d7585983d461c862033c8c2947efc 2023-04-21 01:56:08,898 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/6e2a6d56caad40b2862f776847f25550 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/6e2a6d56caad40b2862f776847f25550 2023-04-21 01:56:08,899 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/9624be29bc084a4eaf689f1273b2ef25 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/9624be29bc084a4eaf689f1273b2ef25 2023-04-21 01:56:08,900 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/376c03460e3841ddb65988776564767d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/376c03460e3841ddb65988776564767d 2023-04-21 01:56:08,901 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1815c310964244689bde97bd3fe045c3 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1815c310964244689bde97bd3fe045c3 2023-04-21 01:56:08,901 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/dfdaf4c55c07453599aa939493dfb8b8 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/dfdaf4c55c07453599aa939493dfb8b8 2023-04-21 01:56:08,902 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/31a5b7cd22394d73aa94ab4a63f27533 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/31a5b7cd22394d73aa94ab4a63f27533 2023-04-21 01:56:08,903 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1f84d2253d48419cae1a7ccea7636f9f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/1f84d2253d48419cae1a7ccea7636f9f 2023-04-21 01:56:08,904 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/25e5260573d94e0b9f284a9452665dbe to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/25e5260573d94e0b9f284a9452665dbe 2023-04-21 01:56:08,905 DEBUG [StoreCloser-TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/b1d35bc16c974f35ac69f2ed0a9425c8 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/b1d35bc16c974f35ac69f2ed0a9425c8 2023-04-21 01:56:08,908 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/recovered.edits/623.seqid, newMaxSeqId=623, maxSeqId=1 2023-04-21 01:56:08,910 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82. 2023-04-21 01:56:08,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 82ecbe07d2c67bb21d6485daedbb8c82: 2023-04-21 01:56:08,911 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:56:08,911 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=82ecbe07d2c67bb21d6485daedbb8c82, regionState=CLOSED 2023-04-21 01:56:08,912 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042168911"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682042168911"}]},"ts":"1682042168911"} 2023-04-21 01:56:08,914 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-21 01:56:08,914 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 82ecbe07d2c67bb21d6485daedbb8c82, server=jenkins-hbase4.apache.org,40131,1682042019399 in 1.5000 sec 2023-04-21 01:56:08,915 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-21 01:56:08,915 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=82ecbe07d2c67bb21d6485daedbb8c82, UNASSIGN in 1.5050 sec 2023-04-21 01:56:08,915 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042168915"}]},"ts":"1682042168915"} 2023-04-21 01:56:08,916 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-21 01:56:08,917 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-21 01:56:08,918 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6110 sec 2023-04-21 01:56:09,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-21 01:56:09,416 INFO [Listener at localhost/39939] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-21 01:56:09,416 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-21 01:56:09,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:56:09,418 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:56:09,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-21 01:56:09,419 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:56:09,422 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:56:09,424 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A, FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B, FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C, FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/recovered.edits] 2023-04-21 01:56:09,426 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/0a215ec942df49f9b90f5d77b95b4e14 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/0a215ec942df49f9b90f5d77b95b4e14 2023-04-21 01:56:09,427 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/8a6cf818d10a48fd925a9f482a8d1cf9 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/8a6cf818d10a48fd925a9f482a8d1cf9 2023-04-21 01:56:09,428 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/f57cf812a4314d19a284d27aecef6d6b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/A/f57cf812a4314d19a284d27aecef6d6b 2023-04-21 01:56:09,430 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/4ef33ee3d8e541528832dc422f9cf31d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/4ef33ee3d8e541528832dc422f9cf31d 2023-04-21 01:56:09,431 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/910b936b4e684fd9b528d2946ac7c44b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/910b936b4e684fd9b528d2946ac7c44b 2023-04-21 01:56:09,432 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/cd9e41d452d94be3ace178d18722b85e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/B/cd9e41d452d94be3ace178d18722b85e 2023-04-21 01:56:09,433 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/484593d709d4492098c8938db62879a3 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/484593d709d4492098c8938db62879a3 2023-04-21 01:56:09,434 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/7269e9c3862942db8a726cfc6ddac223 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/7269e9c3862942db8a726cfc6ddac223 2023-04-21 01:56:09,435 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/7fefe5b96f004de4a9bbc87ae3dff0e7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/C/7fefe5b96f004de4a9bbc87ae3dff0e7 2023-04-21 01:56:09,437 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/recovered.edits/623.seqid to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82/recovered.edits/623.seqid 2023-04-21 01:56:09,437 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/82ecbe07d2c67bb21d6485daedbb8c82 2023-04-21 01:56:09,437 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-21 01:56:09,439 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:56:09,440 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-21 01:56:09,442 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-21 01:56:09,443 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:56:09,443 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-21 01:56:09,443 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682042169443"}]},"ts":"9223372036854775807"} 2023-04-21 01:56:09,444 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-21 01:56:09,444 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 82ecbe07d2c67bb21d6485daedbb8c82, NAME => 'TestAcidGuarantees,,1682042144443.82ecbe07d2c67bb21d6485daedbb8c82.', STARTKEY => '', ENDKEY => ''}] 2023-04-21 01:56:09,444 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-21 01:56:09,444 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682042169444"}]},"ts":"9223372036854775807"} 2023-04-21 01:56:09,445 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-21 01:56:09,447 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:56:09,447 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 31 msec 2023-04-21 01:56:09,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-21 01:56:09,520 INFO [Listener at localhost/39939] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-21 01:56:09,531 INFO [Listener at localhost/39939] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=230 (was 240), OpenFileDescriptor=563 (was 552) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=422 (was 440), ProcessCount=169 (was 169), AvailableMemoryMB=8910 (was 8966) 2023-04-21 01:56:09,543 INFO [Listener at localhost/39939] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=230, OpenFileDescriptor=563, MaxFileDescriptor=60000, SystemLoadAverage=422, ProcessCount=169, AvailableMemoryMB=8910 2023-04-21 01:56:09,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-21 01:56:09,546 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-21 01:56:09,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-21 01:56:09,548 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-21 01:56:09,548 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-21 01:56:09,548 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-21 01:56:09,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-21 01:56:09,550 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:09,550 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89 empty. 2023-04-21 01:56:09,550 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:09,550 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-21 01:56:09,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-21 01:56:09,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-21 01:56:09,956 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-21 01:56:09,958 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 3db499d69ebc07e344a9c8f0976f4e89, NAME => 'TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp 2023-04-21 01:56:10,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-21 01:56:10,363 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-21 01:56:10,364 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 3db499d69ebc07e344a9c8f0976f4e89, disabling compactions & flushes 2023-04-21 01:56:10,364 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:10,364 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:10,364 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. after waiting 0 ms 2023-04-21 01:56:10,364 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:10,364 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:10,364 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:10,366 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-21 01:56:10,366 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042170366"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682042170366"}]},"ts":"1682042170366"} 2023-04-21 01:56:10,367 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-21 01:56:10,368 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-21 01:56:10,368 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042170368"}]},"ts":"1682042170368"} 2023-04-21 01:56:10,369 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-21 01:56:10,372 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3db499d69ebc07e344a9c8f0976f4e89, ASSIGN}] 2023-04-21 01:56:10,374 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3db499d69ebc07e344a9c8f0976f4e89, ASSIGN 2023-04-21 01:56:10,374 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=3db499d69ebc07e344a9c8f0976f4e89, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40131,1682042019399; forceNewPlan=false, retain=false 2023-04-21 01:56:10,525 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=3db499d69ebc07e344a9c8f0976f4e89, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:10,526 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042170525"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682042170525"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682042170525"}]},"ts":"1682042170525"} 2023-04-21 01:56:10,527 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399}] 2023-04-21 01:56:10,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-21 01:56:10,691 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:10,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3db499d69ebc07e344a9c8f0976f4e89, NAME => 'TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.', STARTKEY => '', ENDKEY => ''} 2023-04-21 01:56:10,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:10,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-21 01:56:10,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:10,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:10,693 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:10,694 DEBUG [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A 2023-04-21 01:56:10,694 DEBUG [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A 2023-04-21 01:56:10,695 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:56:10,695 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3db499d69ebc07e344a9c8f0976f4e89 columnFamilyName A 2023-04-21 01:56:10,695 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] regionserver.HStore(310): Store=3db499d69ebc07e344a9c8f0976f4e89/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:56:10,695 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:10,696 DEBUG [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B 2023-04-21 01:56:10,696 DEBUG [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B 2023-04-21 01:56:10,697 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:56:10,697 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3db499d69ebc07e344a9c8f0976f4e89 columnFamilyName B 2023-04-21 01:56:10,697 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] regionserver.HStore(310): Store=3db499d69ebc07e344a9c8f0976f4e89/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:56:10,697 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:10,698 DEBUG [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C 2023-04-21 01:56:10,698 DEBUG [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C 2023-04-21 01:56:10,698 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:56:10,699 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3db499d69ebc07e344a9c8f0976f4e89 columnFamilyName C 2023-04-21 01:56:10,699 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] regionserver.HStore(310): Store=3db499d69ebc07e344a9c8f0976f4e89/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:56:10,699 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:10,700 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:10,700 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:10,702 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-21 01:56:10,703 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:10,704 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-21 01:56:10,705 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 3db499d69ebc07e344a9c8f0976f4e89; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64199007, jitterRate=-0.04336024820804596}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-21 01:56:10,705 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:10,706 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., pid=56, masterSystemTime=1682042170679 2023-04-21 01:56:10,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:10,707 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:10,707 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=3db499d69ebc07e344a9c8f0976f4e89, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:10,707 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042170707"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682042170707"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682042170707"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682042170707"}]},"ts":"1682042170707"} 2023-04-21 01:56:10,709 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-21 01:56:10,709 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 in 181 msec 2023-04-21 01:56:10,710 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-21 01:56:10,711 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3db499d69ebc07e344a9c8f0976f4e89, ASSIGN in 337 msec 2023-04-21 01:56:10,711 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-21 01:56:10,711 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042170711"}]},"ts":"1682042170711"} 2023-04-21 01:56:10,712 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-21 01:56:10,713 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-21 01:56:10,714 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1670 sec 2023-04-21 01:56:11,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-21 01:56:11,653 INFO [Listener at localhost/39939] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-21 01:56:11,654 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x4db22237 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:56:11,659 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66b5b1f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:56:11,660 DEBUG [Listener at localhost/39939] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:56:11,661 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59344, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:56:11,663 DEBUG [Listener at localhost/39939] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-21 01:56:11,665 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38996, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-21 01:56:11,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-21 01:56:11,666 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-21 01:56:11,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-21 01:56:12,076 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-21 01:56:12,077 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-21 01:56:12,077 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-21 01:56:12,079 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-21 01:56:12,082 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3db499d69ebc07e344a9c8f0976f4e89, REOPEN/MOVE}] 2023-04-21 01:56:12,083 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3db499d69ebc07e344a9c8f0976f4e89, REOPEN/MOVE 2023-04-21 01:56:12,083 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=3db499d69ebc07e344a9c8f0976f4e89, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,083 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042172083"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682042172083"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682042172083"}]},"ts":"1682042172083"} 2023-04-21 01:56:12,084 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399}] 2023-04-21 01:56:12,236 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:12,237 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 3db499d69ebc07e344a9c8f0976f4e89, disabling compactions & flushes 2023-04-21 01:56:12,237 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:12,237 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:12,237 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. after waiting 0 ms 2023-04-21 01:56:12,237 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:12,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-21 01:56:12,241 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:12,241 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:12,241 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 3db499d69ebc07e344a9c8f0976f4e89 to self. 2023-04-21 01:56:12,242 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:12,242 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=3db499d69ebc07e344a9c8f0976f4e89, regionState=CLOSED 2023-04-21 01:56:12,242 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042172242"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682042172242"}]},"ts":"1682042172242"} 2023-04-21 01:56:12,244 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-21 01:56:12,244 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 in 159 msec 2023-04-21 01:56:12,245 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=3db499d69ebc07e344a9c8f0976f4e89, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,40131,1682042019399; forceNewPlan=false, retain=true 2023-04-21 01:56:12,395 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=3db499d69ebc07e344a9c8f0976f4e89, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,395 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042172395"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682042172395"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682042172395"}]},"ts":"1682042172395"} 2023-04-21 01:56:12,396 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399}] 2023-04-21 01:56:12,551 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:12,551 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3db499d69ebc07e344a9c8f0976f4e89, NAME => 'TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.', STARTKEY => '', ENDKEY => ''} 2023-04-21 01:56:12,551 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:12,551 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-21 01:56:12,551 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:12,551 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:12,552 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:12,553 DEBUG [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A 2023-04-21 01:56:12,553 DEBUG [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A 2023-04-21 01:56:12,553 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:56:12,554 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3db499d69ebc07e344a9c8f0976f4e89 columnFamilyName A 2023-04-21 01:56:12,555 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] regionserver.HStore(310): Store=3db499d69ebc07e344a9c8f0976f4e89/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:56:12,555 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:12,555 DEBUG [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B 2023-04-21 01:56:12,555 DEBUG [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B 2023-04-21 01:56:12,556 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:56:12,556 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3db499d69ebc07e344a9c8f0976f4e89 columnFamilyName B 2023-04-21 01:56:12,556 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] regionserver.HStore(310): Store=3db499d69ebc07e344a9c8f0976f4e89/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:56:12,556 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:12,557 DEBUG [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C 2023-04-21 01:56:12,557 DEBUG [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C 2023-04-21 01:56:12,557 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-21 01:56:12,557 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3db499d69ebc07e344a9c8f0976f4e89 columnFamilyName C 2023-04-21 01:56:12,557 INFO [StoreOpener-3db499d69ebc07e344a9c8f0976f4e89-1] regionserver.HStore(310): Store=3db499d69ebc07e344a9c8f0976f4e89/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-21 01:56:12,558 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:12,558 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:12,559 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:12,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-21 01:56:12,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:12,562 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 3db499d69ebc07e344a9c8f0976f4e89; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72820582, jitterRate=0.08511123061180115}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-21 01:56:12,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:12,562 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., pid=61, masterSystemTime=1682042172548 2023-04-21 01:56:12,564 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:12,564 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:12,564 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=3db499d69ebc07e344a9c8f0976f4e89, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,564 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042172564"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682042172564"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682042172564"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682042172564"}]},"ts":"1682042172564"} 2023-04-21 01:56:12,566 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-21 01:56:12,566 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 in 169 msec 2023-04-21 01:56:12,567 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-21 01:56:12,567 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3db499d69ebc07e344a9c8f0976f4e89, REOPEN/MOVE in 484 msec 2023-04-21 01:56:12,568 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-21 01:56:12,569 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 488 msec 2023-04-21 01:56:12,570 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 902 msec 2023-04-21 01:56:12,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-21 01:56:12,571 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x08b57aae to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:56:12,576 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@544a4215, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:56:12,576 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x06799455 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:56:12,582 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25baee3e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:56:12,582 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x66c25427 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:56:12,586 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c53bf96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:56:12,587 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x4f4eab6d to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:56:12,593 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b59bdcf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:56:12,593 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x07d997c6 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:56:12,597 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d765193, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:56:12,597 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x32bf16cd to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:56:12,601 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3dca23c0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:56:12,602 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x28164190 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:56:12,605 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62d52321, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:56:12,605 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x308b3f39 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:56:12,609 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a8bf279, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:56:12,609 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x01453d93 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:56:12,614 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68dea875, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:56:12,614 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(139): Connect 0x1cbf09b9 to 127.0.0.1:58383 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-21 01:56:12,620 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34be1618, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-21 01:56:12,629 DEBUG [hconnection-0x5f5ac872-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:56:12,629 DEBUG [hconnection-0x2418c744-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:56:12,632 DEBUG [hconnection-0x2007af25-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:56:12,633 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59386, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:56:12,634 DEBUG [hconnection-0x17a79ce6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:56:12,634 DEBUG [hconnection-0x3551b979-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:56:12,636 DEBUG [hconnection-0x173246c2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:56:12,637 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59424, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:56:12,638 DEBUG [hconnection-0x13a7ccbe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:56:12,638 DEBUG [hconnection-0x52f73bb2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:56:12,638 DEBUG [hconnection-0x30fdd3a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:56:12,639 DEBUG [hconnection-0x4e0bfdb3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-21 01:56:12,641 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59372, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:56:12,642 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59360, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:56:12,642 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59410, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:56:12,642 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59448, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:56:12,642 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59400, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:56:12,643 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59426, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:56:12,643 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59440, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:56:12,644 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:12,651 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59454, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-21 01:56:12,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:12,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:12,656 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:12,656 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:12,657 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:12,657 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:12,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:12,658 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-21 01:56:12,658 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,658 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:12,659 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:12,659 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:12,659 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:12,659 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:12,659 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:12,659 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:12,659 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:12,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:12,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:12,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:12,660 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,660 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:12,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:12,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:12,660 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:12,667 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:12,667 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:12,667 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:12,667 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:12,668 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:12,668 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:12,668 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:12,668 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:12,669 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:12,669 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:12,669 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:12,669 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:12,669 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,669 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:12,669 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:12,670 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:12,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:12,670 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,671 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,671 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:12,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:12,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:12,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:12,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:12,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:12,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e9eced1[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:12,673 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:12,673 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:12,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042232670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:12,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042232671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:12,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59410 deadline: 1682042232674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:12,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59440 deadline: 1682042232674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682042172667, max timestamp=1682042172673Num uniques 859; 2023-04-21 01:56:12,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-21 01:56:12,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682042172667, max timestamp=1682042172673Num uniques 859; 2023-04-21 01:56:12,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682042172667, max timestamp=1682042172673Num uniques 859; 2023-04-21 01:56:12,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-21 01:56:12,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-21 01:56:12,680 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,681 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:12,681 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:12,681 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:12,681 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,681 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:12,682 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:12,682 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:12,682 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:12,682 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:12,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:12,684 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042147d697e10e084987a9ac3d95027dcf5e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042147d697e10e084987a9ac3d95027dcf5e 2023-04-21 01:56:12,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:12,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:12,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:12,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:12,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:12,688 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:12,688 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:12,688 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:12,688 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:12,688 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:12,691 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,691 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:12,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,691 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:12,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:12,692 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:12,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:12,694 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,698 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,698 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:12,698 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:12,698 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:12,698 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:12,698 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:12,698 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:12,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:12,698 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:12,698 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:12,698 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:12,698 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:12,699 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:12,699 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,699 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,704 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/2136a8a0c3d44f65bc38e86695ae8be6 2023-04-21 01:56:12,721 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/f2fe649d59804d4081208a58f0acaf84 2023-04-21 01:56:12,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:12,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682042232720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:12,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:12,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042232775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59410 deadline: 1682042232776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:12,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:12,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59440 deadline: 1682042232776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042232776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,799 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:12,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:12,801 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:12,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:12,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:12,802 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:12,802 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:12,802 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:12,802 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:12,802 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,803 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:12,804 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:12,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:12,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:12,804 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:12,804 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,804 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:12,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:12,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:12,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:12,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:12,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:12,805 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:12,805 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:12,805 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:12,805 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:12,805 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:12,806 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:12,806 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:12,806 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:12,806 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:12,806 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:12,806 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:12,806 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,806 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:12,806 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:12,807 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:12,807 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:12,808 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,808 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:12,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:12,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24276404[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:12,809 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:12,809 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,810 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,810 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:12,810 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:12,810 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:12,810 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,810 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:12,813 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:12,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:12,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:12,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:12,815 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:12,815 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:12,815 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:12,815 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:12,815 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:12,817 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,817 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:12,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,817 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:12,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:12,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:12,818 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:12,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,820 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,820 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:12,820 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:12,821 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:12,821 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,821 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:12,821 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,821 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:12,821 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:12,821 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:12,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:12,821 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:12,821 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,821 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:12,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:12,821 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:12,825 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:12,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682042232824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,921 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:12,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:12,923 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:12,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:12,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:12,925 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:12,925 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:12,925 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:12,925 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:12,925 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,925 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:12,926 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:12,926 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:12,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:12,927 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,927 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:12,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:12,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:12,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:12,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:12,928 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:12,928 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:12,928 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:12,928 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:12,928 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:12,928 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:12,928 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:12,928 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:12,928 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:12,928 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:12,928 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:12,928 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,928 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:12,928 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:12,930 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,931 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:12,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:12,931 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,931 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:12,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:12,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:12,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:12,932 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:12,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@133d7c7[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:12,932 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,933 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,934 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:12,934 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:12,934 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:12,934 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,934 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:12,935 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,935 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:12,935 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:12,935 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:12,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:12,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:12,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:12,937 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:12,937 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:12,937 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:12,937 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:12,937 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:12,939 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,939 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,939 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:12,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:12,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,941 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,944 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:12,944 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:12,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:12,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:12,944 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,944 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:12,944 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:12,944 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:12,944 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:12,944 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,944 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,944 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:12,944 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:12,944 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:12,944 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:12,944 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:12,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:12,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:12,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:12,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59410 deadline: 1682042232979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:12,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042232982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:12,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042232983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:12,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:12,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59440 deadline: 1682042232983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:13,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682042233027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,045 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:13,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:13,047 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:13,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:13,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:13,049 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:13,049 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:13,049 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:13,049 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:13,049 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,049 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,050 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:13,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:13,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:13,051 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,051 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,051 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:13,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:13,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:13,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:13,052 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:13,052 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:13,052 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:13,053 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:13,053 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:13,053 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:13,053 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:13,053 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:13,053 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:13,053 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:13,053 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:13,053 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,053 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:13,053 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:13,055 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,055 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:13,055 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,055 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:13,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:13,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f410577[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:13,057 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:13,058 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,059 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,059 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:13,059 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:13,059 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,059 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:13,059 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:13,063 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,064 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:13,064 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:13,064 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:13,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:13,067 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:13,067 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:13,067 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:13,067 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:13,067 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:13,068 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,068 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:13,068 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:13,069 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:13,070 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:13,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:13,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:13,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:13,073 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:13,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,073 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:13,073 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:13,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:13,073 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:13,073 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:13,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:13,135 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/36cdc0e7440f4c6a83ba597747e6f621 2023-04-21 01:56:13,140 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/2136a8a0c3d44f65bc38e86695ae8be6 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/2136a8a0c3d44f65bc38e86695ae8be6 2023-04-21 01:56:13,144 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/2136a8a0c3d44f65bc38e86695ae8be6, entries=150, sequenceid=15, filesize=25.1 K 2023-04-21 01:56:13,145 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/f2fe649d59804d4081208a58f0acaf84 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f2fe649d59804d4081208a58f0acaf84 2023-04-21 01:56:13,148 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f2fe649d59804d4081208a58f0acaf84, entries=150, sequenceid=15, filesize=11.6 K 2023-04-21 01:56:13,149 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/36cdc0e7440f4c6a83ba597747e6f621 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/36cdc0e7440f4c6a83ba597747e6f621 2023-04-21 01:56:13,154 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/36cdc0e7440f4c6a83ba597747e6f621, entries=150, sequenceid=15, filesize=11.6 K 2023-04-21 01:56:13,155 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 3db499d69ebc07e344a9c8f0976f4e89 in 497ms, sequenceid=15, compaction requested=false 2023-04-21 01:56:13,155 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:13,173 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:13,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:13,175 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:13,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:13,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:13,177 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:13,177 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:13,177 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:13,177 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:13,177 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,177 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,179 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:13,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:13,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:13,179 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,180 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,180 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:13,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:13,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:13,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:13,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:13,181 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:13,181 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:13,181 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:13,184 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:13,184 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:13,184 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:13,184 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:13,184 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-21 01:56:13,184 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:13,184 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:13,184 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:13,184 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:13,184 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:13,184 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:13,204 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304217dff9456a5a64088aa626b964b4c50fa to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304217dff9456a5a64088aa626b964b4c50fa 2023-04-21 01:56:13,225 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/34a17469e14c4b0c82e1b0ae156c3a47 2023-04-21 01:56:13,237 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/0a7825559152432fba94fa5806721d6e 2023-04-21 01:56:13,258 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/bef9d3a04bce490480053b7ae8aa66f5 2023-04-21 01:56:13,262 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/34a17469e14c4b0c82e1b0ae156c3a47 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/34a17469e14c4b0c82e1b0ae156c3a47 2023-04-21 01:56:13,267 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/34a17469e14c4b0c82e1b0ae156c3a47, entries=150, sequenceid=49, filesize=25.1 K 2023-04-21 01:56:13,268 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/0a7825559152432fba94fa5806721d6e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/0a7825559152432fba94fa5806721d6e 2023-04-21 01:56:13,272 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/0a7825559152432fba94fa5806721d6e, entries=150, sequenceid=49, filesize=11.6 K 2023-04-21 01:56:13,272 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/bef9d3a04bce490480053b7ae8aa66f5 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/bef9d3a04bce490480053b7ae8aa66f5 2023-04-21 01:56:13,275 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/bef9d3a04bce490480053b7ae8aa66f5, entries=150, sequenceid=49, filesize=11.6 K 2023-04-21 01:56:13,276 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=0 B/0 for 3db499d69ebc07e344a9c8f0976f4e89 in 92ms, sequenceid=49, compaction requested=false 2023-04-21 01:56:13,276 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:13,276 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:13,276 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:13,276 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:13,276 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,277 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:13,277 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:13,278 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,278 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:13,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:13,279 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,279 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:13,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:13,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:13,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:13,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37e1b408[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:13,281 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:13,281 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,282 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,282 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:13,282 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:13,282 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:13,282 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,282 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:13,285 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,285 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:13,285 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:13,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:13,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:13,285 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:13,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:13,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:13,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:13,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:13,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:13,288 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:13,288 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:13,288 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:13,288 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:13,289 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:13,292 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,292 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:13,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,292 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:13,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:13,293 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:13,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,294 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:13,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,296 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,296 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,296 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,296 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:13,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:13,296 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:13,296 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:13,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:13,296 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:13,297 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:13,297 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:13,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:13,297 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:13,297 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:13,297 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,297 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:13,297 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:13,307 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-21 01:56:13,307 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:13,308 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:13,308 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:13,308 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:13,308 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:13,308 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:13,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:13,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042233332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042173330, max timestamp=1682042173335Num uniques 859; 2023-04-21 01:56:13,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042173330, max timestamp=1682042173335Num uniques 859; 2023-04-21 01:56:13,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:13,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:13,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042173330, max timestamp=1682042173335Num uniques 859; 2023-04-21 01:56:13,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:13,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:13,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042233333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,354 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421dae349e8acfd4178851e5efce46e9497 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421dae349e8acfd4178851e5efce46e9497 2023-04-21 01:56:13,360 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:13,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682042233358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:13,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:13,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59440 deadline: 1682042233360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59410 deadline: 1682042233360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,397 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:13,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:13,401 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:13,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:13,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:13,404 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:13,404 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:13,404 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:13,404 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:13,405 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,405 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,406 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,406 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:13,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:13,407 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,407 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:13,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:13,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:13,414 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:13,414 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:13,414 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:13,414 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:13,414 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:13,414 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:13,414 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:13,414 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:13,414 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:13,414 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:13,414 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:13,414 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,414 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:13,414 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:13,416 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,416 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:13,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:13,416 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,416 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:13,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:13,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:13,418 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:13,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2879e1b4[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:13,418 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,420 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,420 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,420 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:13,420 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:13,420 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:13,420 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:13,422 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,422 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:13,422 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:13,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:13,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:13,422 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:13,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:13,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:13,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:13,424 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:13,425 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:13,425 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:13,425 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:13,425 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:13,426 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,426 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,426 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:13,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:13,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:13,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:13,427 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:13,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:13,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:13,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,428 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:13,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,431 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:13,431 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:13,431 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:13,431 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,431 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,431 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:13,431 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:13,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:13,431 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,431 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,431 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:13,431 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:13,432 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:13,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:13,431 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,432 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:13,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042233438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:13,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042233439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:13,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682042233461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:13,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59410 deadline: 1682042233466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:13,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59440 deadline: 1682042233466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,532 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:13,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:13,535 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:13,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:13,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:13,537 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:13,537 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:13,537 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:13,537 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:13,537 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,537 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,539 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,539 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:13,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:13,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:13,539 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,539 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:13,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:13,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,540 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:13,541 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:13,541 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:13,541 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:13,541 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:13,541 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:13,541 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:13,541 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:13,541 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:13,541 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:13,541 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:13,541 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:13,541 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:13,541 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:13,541 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,541 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:13,542 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:13,543 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,543 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:13,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:13,543 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,543 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:13,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:13,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:13,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:13,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:13,545 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:13,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37eb8693[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:13,545 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,546 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,546 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:13,546 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,546 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:13,546 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:13,546 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:13,547 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,547 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:13,547 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:13,547 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:13,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:13,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:13,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:13,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:13,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:13,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:13,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:13,550 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:13,550 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:13,550 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:13,550 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:13,550 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:13,552 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:13,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:13,553 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,553 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:13,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:13,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,553 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:13,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:13,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:13,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,555 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:13,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,562 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,562 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,562 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,562 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,562 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:13,562 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:13,562 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,562 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:13,562 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:13,562 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:13,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:13,562 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:13,563 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:13,563 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:13,562 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:13,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:13,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:13,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042233643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:13,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042233643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,663 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:13,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:13,665 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:13,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:13,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:13,666 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:13,666 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:13,666 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:13,666 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:13,667 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,667 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:13,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682042233665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,670 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,670 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:13,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:13,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:13,670 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,670 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:13,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:13,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:13,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:13,671 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:13,672 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:13,672 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:13,672 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:13,672 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:13,672 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:13,672 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:13,672 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:13,672 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:13,672 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:13,672 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:13,672 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,672 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:13,672 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:13,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:13,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59410 deadline: 1682042233670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,674 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,674 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:13,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:13,674 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,674 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:13,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:13,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:13,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59440 deadline: 1682042233671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:13,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:13,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:13,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@407657a4[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:13,676 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:13,676 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,677 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,677 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,677 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:13,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,678 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:13,678 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:13,678 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:13,682 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,682 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:13,682 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:13,682 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:13,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:13,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:13,685 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:13,685 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:13,685 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:13,685 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:13,685 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:13,687 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,687 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:13,687 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:13,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:13,687 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:13,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,689 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:13,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,697 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,697 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:13,697 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:13,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:13,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:13,697 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,697 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:13,697 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:13,698 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:13,697 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,698 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,698 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,698 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,698 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:13,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:13,698 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:13,698 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:13,772 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=60, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/f3785e1b685b4e5e9b3abdde7f023aea 2023-04-21 01:56:13,798 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:13,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:13,800 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:13,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:13,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:13,802 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:13,802 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:13,802 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:13,802 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:13,802 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,802 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,803 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,803 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:13,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:13,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:13,804 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,804 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:13,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:13,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:13,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:13,806 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:13,806 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:13,806 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:13,806 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:13,806 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:13,806 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:13,806 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:13,806 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:13,806 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:13,806 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:13,806 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:13,806 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,806 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:13,806 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:13,809 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,809 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:13,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:13,810 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,810 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:13,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:13,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:13,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:13,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:13,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a8dfa56[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:13,811 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:13,811 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,812 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:13,813 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,813 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:13,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:13,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:13,814 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:13,814 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:13,814 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:13,814 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:13,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:13,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:13,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:13,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:13,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:13,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:13,817 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:13,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:13,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:13,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:13,817 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:13,818 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,818 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:13,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,819 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:13,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:13,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:13,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:13,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:13,820 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:13,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:13,821 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,824 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:13,824 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:13,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:13,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:13,824 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:13,825 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:13,824 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,824 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:13,825 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,825 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,825 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,825 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:13,825 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:13,825 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:13,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:13,825 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,825 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,925 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:13,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:13,927 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:13,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:13,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:13,929 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:13,929 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:13,929 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:13,929 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:13,929 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,929 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,930 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:13,930 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:13,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:13,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,931 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,931 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:13,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:13,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:13,932 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:13,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:13,933 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:13,933 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:13,933 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:13,937 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:13,937 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:13,937 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:13,937 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:13,937 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:13,937 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:13,937 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:13,937 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:13,937 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,937 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:13,937 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:13,939 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,939 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:13,939 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,939 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:13,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:13,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d1ec0f2[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:13,940 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:13,940 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,941 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,941 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:13,941 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:13,941 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:13,942 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,942 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:13,944 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,944 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:13,944 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:13,944 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:13,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:13,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:13,946 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:13,946 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:13,946 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:13,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:13,946 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:13,946 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:13,947 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:13,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042233945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:13,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042233947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,949 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,949 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,949 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:13,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:13,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:13,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:13,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:13,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:13,950 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,951 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,953 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,953 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:13,953 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:13,954 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:13,954 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:13,954 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:13,954 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:13,954 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,954 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:13,954 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:13,954 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:13,954 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:13,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:13,954 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:13,954 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:13,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:13,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682042233971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:13,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59410 deadline: 1682042233976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:13,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:13,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59440 deadline: 1682042233977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,054 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:14,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:14,056 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:14,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:14,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:14,057 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:14,057 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:14,057 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:14,057 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:14,058 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,058 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:14,060 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:14,060 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:14,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:14,061 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:14,061 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:14,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:14,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:14,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:14,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:14,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:14,062 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:14,062 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:14,062 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:14,062 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:14,062 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:14,062 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:14,062 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:14,062 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:14,062 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:14,062 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:14,062 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:14,062 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,062 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:14,062 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:14,064 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,064 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:14,064 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,064 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:14,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:14,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76aa12dc[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:14,065 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:14,065 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,066 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,066 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:14,066 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:14,066 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:14,066 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:14,067 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,067 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:14,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:14,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:14,067 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:14,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:14,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:14,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:14,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:14,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:14,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:14,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:14,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:14,069 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:14,070 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:14,070 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:14,070 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:14,070 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:14,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,073 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:14,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:14,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:14,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:14,074 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,076 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,077 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:14,077 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:14,077 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:14,077 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:14,078 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,078 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:14,078 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:14,077 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:14,078 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,077 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:14,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:14,078 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,078 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,078 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:14,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:14,078 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:14,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:14,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:14,178 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:14,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:14,180 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:14,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:14,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:14,181 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:14,181 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:14,181 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:14,181 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:14,182 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,182 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:14,183 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:14,183 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:14,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:14,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:14,183 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,183 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:14,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:14,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:14,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:14,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:14,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:14,184 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:14,184 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:14,184 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:14,185 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:14,185 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:14,185 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:14,185 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:14,185 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:14,185 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:14,185 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:14,185 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:14,185 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,185 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:14,185 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:14,185 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/88aee48cdfd84ad6892ca1c284da111e 2023-04-21 01:56:14,187 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,187 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:14,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:14,187 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,187 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:14,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:14,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@413b0da[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:14,190 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:14,190 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,192 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:14,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:14,192 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:14,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,193 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:14,194 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,194 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:14,194 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:14,194 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:14,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:14,195 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/a5402c1baa19407ea0cb4634ec74aa1f 2023-04-21 01:56:14,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:14,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:14,197 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:14,198 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:14,198 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:14,198 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:14,198 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:14,199 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/f3785e1b685b4e5e9b3abdde7f023aea as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/f3785e1b685b4e5e9b3abdde7f023aea 2023-04-21 01:56:14,199 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,199 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:14,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,199 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:14,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:14,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:14,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:14,200 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:14,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:14,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:14,202 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:14,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:14,203 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/f3785e1b685b4e5e9b3abdde7f023aea, entries=400, sequenceid=60, filesize=59.0 K 2023-04-21 01:56:14,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,204 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/88aee48cdfd84ad6892ca1c284da111e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/88aee48cdfd84ad6892ca1c284da111e 2023-04-21 01:56:14,204 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,205 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:14,205 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:14,205 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,205 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,205 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,205 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:14,205 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:14,205 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:14,205 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:14,205 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:14,205 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:14,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:14,205 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:14,205 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:14,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:14,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:14,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:14,207 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/88aee48cdfd84ad6892ca1c284da111e, entries=150, sequenceid=60, filesize=11.6 K 2023-04-21 01:56:14,208 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/a5402c1baa19407ea0cb4634ec74aa1f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/a5402c1baa19407ea0cb4634ec74aa1f 2023-04-21 01:56:14,211 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/a5402c1baa19407ea0cb4634ec74aa1f, entries=150, sequenceid=60, filesize=11.6 K 2023-04-21 01:56:14,211 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 3db499d69ebc07e344a9c8f0976f4e89 in 904ms, sequenceid=60, compaction requested=true 2023-04-21 01:56:14,211 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:14,212 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:14,212 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:56:14,212 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:14,212 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:56:14,212 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:14,213 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 111946 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:56:14,213 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:56:14,213 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/A is initiating minor compaction (all files) 2023-04-21 01:56:14,213 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/B is initiating minor compaction (all files) 2023-04-21 01:56:14,213 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/A in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:14,213 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/B in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:14,213 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/2136a8a0c3d44f65bc38e86695ae8be6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/34a17469e14c4b0c82e1b0ae156c3a47, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/f3785e1b685b4e5e9b3abdde7f023aea] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=109.3 K 2023-04-21 01:56:14,213 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f2fe649d59804d4081208a58f0acaf84, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/0a7825559152432fba94fa5806721d6e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/88aee48cdfd84ad6892ca1c284da111e] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=34.8 K 2023-04-21 01:56:14,213 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 2136a8a0c3d44f65bc38e86695ae8be6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682042172655 2023-04-21 01:56:14,213 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting f2fe649d59804d4081208a58f0acaf84, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682042172655 2023-04-21 01:56:14,214 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 0a7825559152432fba94fa5806721d6e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682042172708 2023-04-21 01:56:14,214 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 34a17469e14c4b0c82e1b0ae156c3a47, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682042172708 2023-04-21 01:56:14,214 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 88aee48cdfd84ad6892ca1c284da111e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682042173300 2023-04-21 01:56:14,214 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting f3785e1b685b4e5e9b3abdde7f023aea, keycount=400, bloomtype=ROW, size=59.0 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682042173297 2023-04-21 01:56:14,221 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#B#compaction#542 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:14,225 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#A#compaction#543 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:14,305 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:14,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:14,307 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:14,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:14,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:14,308 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:14,309 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:14,309 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:14,309 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:14,309 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,309 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:14,312 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:14,312 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:14,312 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,312 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:14,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:14,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:14,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:14,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:14,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:14,313 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:14,313 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:14,313 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:14,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:14,314 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:14,314 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:14,314 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:14,314 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-21 01:56:14,314 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:14,314 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:14,314 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:14,314 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:14,314 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:14,314 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:14,438 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-21 01:56:14,453 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:14,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:14,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:14,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042234467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:14,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042234469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:14,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682042234475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:14,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59410 deadline: 1682042234481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:14,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59440 deadline: 1682042234482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:14,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042234573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:14,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042234576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,631 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/2876bbc1114d4bbf9fa33300143c0db9 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/2876bbc1114d4bbf9fa33300143c0db9 2023-04-21 01:56:14,635 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/B of 3db499d69ebc07e344a9c8f0976f4e89 into 2876bbc1114d4bbf9fa33300143c0db9(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:14,636 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:14,636 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/B, priority=13, startTime=1682042174212; duration=0sec 2023-04-21 01:56:14,636 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:14,636 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:56:14,637 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:56:14,637 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/C is initiating minor compaction (all files) 2023-04-21 01:56:14,638 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/C in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:14,638 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/36cdc0e7440f4c6a83ba597747e6f621, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/bef9d3a04bce490480053b7ae8aa66f5, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/a5402c1baa19407ea0cb4634ec74aa1f] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=34.8 K 2023-04-21 01:56:14,638 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 36cdc0e7440f4c6a83ba597747e6f621, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682042172655 2023-04-21 01:56:14,639 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting bef9d3a04bce490480053b7ae8aa66f5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682042172708 2023-04-21 01:56:14,639 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting a5402c1baa19407ea0cb4634ec74aa1f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682042173300 2023-04-21 01:56:14,648 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#C#compaction#545 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:14,650 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/88484729d9ec4ce9a507c59660d63a60 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/88484729d9ec4ce9a507c59660d63a60 2023-04-21 01:56:14,655 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/A of 3db499d69ebc07e344a9c8f0976f4e89 into 88484729d9ec4ce9a507c59660d63a60(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:14,655 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:14,655 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/A, priority=13, startTime=1682042174212; duration=0sec 2023-04-21 01:56:14,655 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:14,729 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042122e828da8fdf444eb1c00dae411c2c39 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042122e828da8fdf444eb1c00dae411c2c39 2023-04-21 01:56:14,736 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=96, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/c7c5e5dc96bc43f69c302ccd8b08f43c 2023-04-21 01:56:14,747 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/b05289a8617d4fd3b179dea8ba4a2964 2023-04-21 01:56:14,760 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/e444ac010dfa43508735285f7852a9ea 2023-04-21 01:56:14,764 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/c7c5e5dc96bc43f69c302ccd8b08f43c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/c7c5e5dc96bc43f69c302ccd8b08f43c 2023-04-21 01:56:14,768 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/c7c5e5dc96bc43f69c302ccd8b08f43c, entries=150, sequenceid=96, filesize=25.1 K 2023-04-21 01:56:14,769 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/b05289a8617d4fd3b179dea8ba4a2964 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b05289a8617d4fd3b179dea8ba4a2964 2023-04-21 01:56:14,773 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b05289a8617d4fd3b179dea8ba4a2964, entries=150, sequenceid=96, filesize=11.6 K 2023-04-21 01:56:14,774 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/e444ac010dfa43508735285f7852a9ea as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/e444ac010dfa43508735285f7852a9ea 2023-04-21 01:56:14,781 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/e444ac010dfa43508735285f7852a9ea, entries=150, sequenceid=96, filesize=11.6 K 2023-04-21 01:56:14,782 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 3db499d69ebc07e344a9c8f0976f4e89 in 468ms, sequenceid=96, compaction requested=false 2023-04-21 01:56:14,782 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:14,782 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:14,782 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:14,782 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:14,782 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,782 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:14,782 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:14,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:14,784 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-21 01:56:14,784 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,785 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:14,785 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:14,786 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,786 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:14,785 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:14,786 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:14,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:14,786 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:14,786 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:14,786 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:14,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:14,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:14,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1159b544[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:14,787 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:14,787 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,789 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,789 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:14,789 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:14,789 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:14,789 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,789 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:14,790 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:14,791 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:14,791 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:14,791 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:14,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:14,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:14,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:14,793 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:14,793 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:14,793 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:14,794 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:14,794 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:14,797 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,797 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:14,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:14,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,797 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:14,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:14,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:14,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:14,798 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:14,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:14,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:14,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:14,800 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:14,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,803 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,803 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:14,803 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:14,803 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:14,803 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:14,803 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:14,803 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:14,803 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:14,803 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,803 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,803 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:14,803 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:14,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:14,803 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:14,803 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:14,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:14,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:14,813 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304214cd63ed0a16b42ccaed973328952a37e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304214cd63ed0a16b42ccaed973328952a37e 2023-04-21 01:56:14,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:14,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042234812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:14,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042234813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,818 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=111, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/d374075ee3084089b8a6733b0ccb7868 2023-04-21 01:56:14,828 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/6cea4d6900984fc39da87ff38dbb43ef 2023-04-21 01:56:14,841 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/12e49110ccca498b908c5e24b2c2c597 2023-04-21 01:56:14,846 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/d374075ee3084089b8a6733b0ccb7868 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/d374075ee3084089b8a6733b0ccb7868 2023-04-21 01:56:14,850 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/d374075ee3084089b8a6733b0ccb7868, entries=200, sequenceid=111, filesize=31.9 K 2023-04-21 01:56:14,851 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/6cea4d6900984fc39da87ff38dbb43ef as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/6cea4d6900984fc39da87ff38dbb43ef 2023-04-21 01:56:14,854 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/6cea4d6900984fc39da87ff38dbb43ef, entries=150, sequenceid=111, filesize=11.6 K 2023-04-21 01:56:14,855 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/12e49110ccca498b908c5e24b2c2c597 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/12e49110ccca498b908c5e24b2c2c597 2023-04-21 01:56:14,857 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/12e49110ccca498b908c5e24b2c2c597, entries=150, sequenceid=111, filesize=11.6 K 2023-04-21 01:56:14,858 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 3db499d69ebc07e344a9c8f0976f4e89 in 74ms, sequenceid=111, compaction requested=true 2023-04-21 01:56:14,858 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:14,858 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:14,858 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:14,858 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:56:14,858 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-21 01:56:14,859 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84281 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:56:14,859 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/A is initiating minor compaction (all files) 2023-04-21 01:56:14,859 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/A in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:14,859 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/88484729d9ec4ce9a507c59660d63a60, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/c7c5e5dc96bc43f69c302ccd8b08f43c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/d374075ee3084089b8a6733b0ccb7868] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=82.3 K 2023-04-21 01:56:14,860 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 88484729d9ec4ce9a507c59660d63a60, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682042173300 2023-04-21 01:56:14,860 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting c7c5e5dc96bc43f69c302ccd8b08f43c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1682042173356 2023-04-21 01:56:14,861 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting d374075ee3084089b8a6733b0ccb7868, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682042174465 2023-04-21 01:56:14,873 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#A#compaction#551 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:14,903 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:14,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:14,906 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:14,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:14,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:14,907 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:14,907 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:14,907 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:14,907 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:14,908 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,908 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:14,912 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,912 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:14,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:14,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:14,912 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,912 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:14,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:14,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:14,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:14,913 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:14,913 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:14,913 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:14,914 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:14,914 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:14,914 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:14,914 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:14,914 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:14,914 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:14,914 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-21 01:56:14,914 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:14,914 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:14,914 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:14,914 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:14,915 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:14,915 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:14,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:14,919 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:14,931 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304218680123a6de34525b68241d57de906a3 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304218680123a6de34525b68241d57de906a3 2023-04-21 01:56:14,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:14,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042234932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:14,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042234933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:14,938 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=130, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/2820906420184a35926a6c4ef1185a4a 2023-04-21 01:56:14,949 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/3aadde79b4044f91abbca813c788b998 2023-04-21 01:56:15,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:15,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042235034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:15,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042235037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,062 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/3da7a0fb83064751b5a85f3912b70adf as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/3da7a0fb83064751b5a85f3912b70adf 2023-04-21 01:56:15,066 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/C of 3db499d69ebc07e344a9c8f0976f4e89 into 3da7a0fb83064751b5a85f3912b70adf(size=11.7 K), total size for store is 34.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:15,066 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:15,066 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/C, priority=13, startTime=1682042174212; duration=0sec 2023-04-21 01:56:15,066 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-21 01:56:15,066 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:56:15,068 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:56:15,068 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/B is initiating minor compaction (all files) 2023-04-21 01:56:15,068 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/B in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:15,068 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/2876bbc1114d4bbf9fa33300143c0db9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b05289a8617d4fd3b179dea8ba4a2964, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/6cea4d6900984fc39da87ff38dbb43ef] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=34.9 K 2023-04-21 01:56:15,068 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 2876bbc1114d4bbf9fa33300143c0db9, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682042173300 2023-04-21 01:56:15,069 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting b05289a8617d4fd3b179dea8ba4a2964, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1682042173356 2023-04-21 01:56:15,069 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 6cea4d6900984fc39da87ff38dbb43ef, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682042174467 2023-04-21 01:56:15,074 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#B#compaction#555 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:15,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:15,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042235238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:15,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042235242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,289 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/9950c940a6bf4605bb8658dda72447e8 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/9950c940a6bf4605bb8658dda72447e8 2023-04-21 01:56:15,293 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/A of 3db499d69ebc07e344a9c8f0976f4e89 into 9950c940a6bf4605bb8658dda72447e8(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:15,293 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:15,293 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/A, priority=13, startTime=1682042174858; duration=0sec 2023-04-21 01:56:15,293 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:15,293 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:56:15,295 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:56:15,295 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/C is initiating minor compaction (all files) 2023-04-21 01:56:15,295 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/C in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:15,295 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/3da7a0fb83064751b5a85f3912b70adf, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/e444ac010dfa43508735285f7852a9ea, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/12e49110ccca498b908c5e24b2c2c597] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=34.9 K 2023-04-21 01:56:15,295 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 3da7a0fb83064751b5a85f3912b70adf, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682042173300 2023-04-21 01:56:15,296 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting e444ac010dfa43508735285f7852a9ea, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1682042173356 2023-04-21 01:56:15,296 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 12e49110ccca498b908c5e24b2c2c597, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682042174467 2023-04-21 01:56:15,302 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#C#compaction#556 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:15,358 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/c471342b3c4f46a78addb7e03967bd9f 2023-04-21 01:56:15,362 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/2820906420184a35926a6c4ef1185a4a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/2820906420184a35926a6c4ef1185a4a 2023-04-21 01:56:15,365 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/2820906420184a35926a6c4ef1185a4a, entries=150, sequenceid=130, filesize=25.2 K 2023-04-21 01:56:15,366 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/3aadde79b4044f91abbca813c788b998 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/3aadde79b4044f91abbca813c788b998 2023-04-21 01:56:15,369 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/3aadde79b4044f91abbca813c788b998, entries=150, sequenceid=130, filesize=11.7 K 2023-04-21 01:56:15,370 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/c471342b3c4f46a78addb7e03967bd9f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/c471342b3c4f46a78addb7e03967bd9f 2023-04-21 01:56:15,373 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/c471342b3c4f46a78addb7e03967bd9f, entries=150, sequenceid=130, filesize=11.7 K 2023-04-21 01:56:15,374 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 3db499d69ebc07e344a9c8f0976f4e89 in 460ms, sequenceid=130, compaction requested=false 2023-04-21 01:56:15,374 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:15,374 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:15,374 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:15,374 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:15,374 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,374 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:15,375 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:15,376 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,376 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:15,377 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,377 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:15,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:15,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a2beecd[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:15,378 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:15,378 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,379 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,379 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:15,379 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:15,379 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:15,379 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,379 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:15,380 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,380 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:15,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:15,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:15,380 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:15,381 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:15,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:15,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:15,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:15,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:15,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:15,383 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:15,383 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:15,383 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:15,383 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:15,383 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:15,386 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,386 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:15,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:15,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,386 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:15,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:15,387 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:15,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:15,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:15,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,388 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:15,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,390 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:15,390 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:15,390 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:15,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:15,390 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:15,391 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:15,390 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:15,391 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:15,391 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,391 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,391 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,391 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,391 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:15,390 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:15,391 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:15,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:15,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:15,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:15,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:15,489 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-21 01:56:15,489 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:15,489 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:15,489 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:15,489 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:15,489 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:15,489 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:15,490 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/b6553f2485594a2c9d66fa3a17f32af4 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b6553f2485594a2c9d66fa3a17f32af4 2023-04-21 01:56:15,491 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:15,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:15,494 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:15,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:15,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:15,495 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:15,495 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:15,495 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:15,495 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:15,496 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,496 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:15,497 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:15,497 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:15,497 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/B of 3db499d69ebc07e344a9c8f0976f4e89 into b6553f2485594a2c9d66fa3a17f32af4(size=11.8 K), total size for store is 23.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:15,497 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:15,497 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/B, priority=13, startTime=1682042174858; duration=0sec 2023-04-21 01:56:15,497 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:15,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:15,498 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,498 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:15,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:15,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:15,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:15,499 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:15,499 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:15,499 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:15,499 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:15,499 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:15,499 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:15,499 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:15,499 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:15,499 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:15,499 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:15,500 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:15,500 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:15,500 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,500 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:15,500 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:15,501 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,501 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:15,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:15,502 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,502 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:15,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:15,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:15,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:15,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:15,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cb13fa6[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:15,504 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:15,504 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,507 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,507 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:15,507 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,507 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:15,507 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:15,507 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:15,508 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:15,508 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:15,508 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:15,508 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:15,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:15,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:15,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:15,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,510 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304211a205c12e6be4065b2f94c14c6b49ab5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304211a205c12e6be4065b2f94c14c6b49ab5 2023-04-21 01:56:15,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:15,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:15,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:15,512 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:15,512 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:15,512 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:15,512 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:15,512 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:15,514 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:15,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59410 deadline: 1682042235509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:15,514 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59440 deadline: 1682042235510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,514 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:15,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:15,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682042235510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:15,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:15,515 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:15,515 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=146, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/51536db93ee444e9a51b3f8a5751b0fa 2023-04-21 01:56:15,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:15,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:15,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:15,517 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:15,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,523 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:15,523 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:15,523 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:15,524 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,524 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:15,524 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:15,524 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,524 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,524 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:15,524 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:15,524 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:15,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:15,524 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:15,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:15,524 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:15,524 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:15,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:15,526 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/7b6ebf979a4b432599c0c572d5b5d5bf 2023-04-21 01:56:15,536 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/d3a13ace148e4b9eb477b3aae8a94f13 2023-04-21 01:56:15,540 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/51536db93ee444e9a51b3f8a5751b0fa as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/51536db93ee444e9a51b3f8a5751b0fa 2023-04-21 01:56:15,544 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/51536db93ee444e9a51b3f8a5751b0fa, entries=150, sequenceid=146, filesize=25.3 K 2023-04-21 01:56:15,544 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/7b6ebf979a4b432599c0c572d5b5d5bf as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/7b6ebf979a4b432599c0c572d5b5d5bf 2023-04-21 01:56:15,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:15,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042235542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:15,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042235544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,549 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/7b6ebf979a4b432599c0c572d5b5d5bf, entries=150, sequenceid=146, filesize=11.8 K 2023-04-21 01:56:15,549 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/d3a13ace148e4b9eb477b3aae8a94f13 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/d3a13ace148e4b9eb477b3aae8a94f13 2023-04-21 01:56:15,552 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/d3a13ace148e4b9eb477b3aae8a94f13, entries=150, sequenceid=146, filesize=11.8 K 2023-04-21 01:56:15,553 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 3db499d69ebc07e344a9c8f0976f4e89 in 64ms, sequenceid=146, compaction requested=true 2023-04-21 01:56:15,553 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:15,553 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:15,553 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:15,553 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:56:15,554 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-21 01:56:15,560 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77644 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:56:15,560 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/A is initiating minor compaction (all files) 2023-04-21 01:56:15,560 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/A in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:15,560 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/9950c940a6bf4605bb8658dda72447e8, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/2820906420184a35926a6c4ef1185a4a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/51536db93ee444e9a51b3f8a5751b0fa] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=75.8 K 2023-04-21 01:56:15,561 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 9950c940a6bf4605bb8658dda72447e8, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682042174467 2023-04-21 01:56:15,561 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 2820906420184a35926a6c4ef1185a4a, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1682042174806 2023-04-21 01:56:15,562 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 51536db93ee444e9a51b3f8a5751b0fa, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1682042174930 2023-04-21 01:56:15,579 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#A#compaction#560 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:15,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:15,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1682042175503, max timestamp=1682042175617Num uniques 859; 2023-04-21 01:56:15,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1682042175503, max timestamp=1682042175617Num uniques 859; 2023-04-21 01:56:15,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-21 01:56:15,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-21 01:56:15,623 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=67.09 KB heapSize=184.48 KB 2023-04-21 01:56:15,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682042175506, max timestamp=1682042175617Num uniques 859; 2023-04-21 01:56:15,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:56:15,623 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:15,623 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:15,623 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:15,623 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:15,623 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:15,624 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:15,624 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:15,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:15,627 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:15,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:15,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:15,636 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:15,636 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:15,636 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:15,636 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:15,637 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,637 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:15,642 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,642 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:15,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:15,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:15,642 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,642 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:15,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:15,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:15,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:15,643 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:15,645 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:15,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:15,645 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:15,645 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:15,645 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:15,646 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:15,646 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:15,646 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:15,646 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:15,646 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:15,646 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:15,646 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:15,646 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:15,646 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,646 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:15,646 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:15,648 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,648 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:15,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:15,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:15,649 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,649 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:15,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:15,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:15,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:15,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@453dc18a[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:15,651 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:15,651 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,652 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,652 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:15,652 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:15,652 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:15,653 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,653 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:15,654 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:15,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:15,654 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:15,654 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:15,654 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:15,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:15,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:15,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:15,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042175636, max timestamp=1682042175654Num uniques 859; 2023-04-21 01:56:15,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042175636, max timestamp=1682042175654Num uniques 859; 2023-04-21 01:56:15,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:15,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042175636, max timestamp=1682042175654Num uniques 859; 2023-04-21 01:56:15,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:15,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:15,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:15,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:15,657 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:15,657 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:15,657 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:15,657 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:15,657 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:15,660 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,661 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,661 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:15,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:15,661 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:15,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:15,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,664 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:15,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:15,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:15,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,666 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:15,667 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:15,667 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:15,667 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:15,667 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:15,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:15,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:15,667 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:15,667 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:15,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:15,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:15,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:15,669 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421f109f03103074365b610e796aad55a79 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421f109f03103074365b610e796aad55a79 2023-04-21 01:56:15,687 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=168, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/da6d97a6aebd4a9f94c11ad293da892f 2023-04-21 01:56:15,701 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/336e9840de8d4df5b5f76cfa03702d7a 2023-04-21 01:56:15,712 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/66aeca9ed0b949efa8d36c7e6ec7c7fe 2023-04-21 01:56:15,716 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/64d25ca4b84d41fdb00981cc29d87ebe as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/64d25ca4b84d41fdb00981cc29d87ebe 2023-04-21 01:56:15,717 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/da6d97a6aebd4a9f94c11ad293da892f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/da6d97a6aebd4a9f94c11ad293da892f 2023-04-21 01:56:15,726 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/C of 3db499d69ebc07e344a9c8f0976f4e89 into 64d25ca4b84d41fdb00981cc29d87ebe(size=11.8 K), total size for store is 35.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:15,726 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:15,726 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/C, priority=13, startTime=1682042174858; duration=0sec 2023-04-21 01:56:15,726 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-21 01:56:15,726 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:56:15,727 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/da6d97a6aebd4a9f94c11ad293da892f, entries=150, sequenceid=168, filesize=25.3 K 2023-04-21 01:56:15,727 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36064 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:56:15,727 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/B is initiating minor compaction (all files) 2023-04-21 01:56:15,727 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/B in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:15,727 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b6553f2485594a2c9d66fa3a17f32af4, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/3aadde79b4044f91abbca813c788b998, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/7b6ebf979a4b432599c0c572d5b5d5bf] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=35.2 K 2023-04-21 01:56:15,728 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting b6553f2485594a2c9d66fa3a17f32af4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682042174467 2023-04-21 01:56:15,728 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/336e9840de8d4df5b5f76cfa03702d7a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/336e9840de8d4df5b5f76cfa03702d7a 2023-04-21 01:56:15,728 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 3aadde79b4044f91abbca813c788b998, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1682042174806 2023-04-21 01:56:15,729 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 7b6ebf979a4b432599c0c572d5b5d5bf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1682042174930 2023-04-21 01:56:15,732 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/336e9840de8d4df5b5f76cfa03702d7a, entries=150, sequenceid=168, filesize=11.8 K 2023-04-21 01:56:15,733 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/66aeca9ed0b949efa8d36c7e6ec7c7fe as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/66aeca9ed0b949efa8d36c7e6ec7c7fe 2023-04-21 01:56:15,735 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#B#compaction#564 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:15,736 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/66aeca9ed0b949efa8d36c7e6ec7c7fe, entries=150, sequenceid=168, filesize=11.8 K 2023-04-21 01:56:15,737 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~38.02 KB/38930, heapSize ~91.58 KB/93776, currentSize=140.89 KB/144270 for 3db499d69ebc07e344a9c8f0976f4e89 in 114ms, sequenceid=168, compaction requested=true 2023-04-21 01:56:15,737 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:15,737 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-21 01:56:15,737 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-21 01:56:15,737 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-21 01:56:15,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682042175723, max timestamp=1682042175735Num uniques 1018; 2023-04-21 01:56:15,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:15,738 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=116.29 KB heapSize=332.05 KB 2023-04-21 01:56:15,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682042175723, max timestamp=1682042175735Num uniques 1018; 2023-04-21 01:56:15,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-21 01:56:15,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-21 01:56:15,738 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:15,738 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:15,738 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:15,738 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-21 01:56:15,738 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:15,738 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:15,745 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/f5e94ad539eb4438ba013adbfacebf35 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f5e94ad539eb4438ba013adbfacebf35 2023-04-21 01:56:15,750 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304217171015d39e54a9e9c1a61b3dafd30ec to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304217171015d39e54a9e9c1a61b3dafd30ec 2023-04-21 01:56:15,751 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/B of 3db499d69ebc07e344a9c8f0976f4e89 into f5e94ad539eb4438ba013adbfacebf35(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:15,751 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:15,751 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/B, priority=13, startTime=1682042175553; duration=0sec 2023-04-21 01:56:15,751 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-21 01:56:15,751 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 3 compacting, 1 eligible, 16 blocking 2023-04-21 01:56:15,751 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-21 01:56:15,751 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-21 01:56:15,752 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. because compaction request was cancelled 2023-04-21 01:56:15,752 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:56:15,753 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48100 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:56:15,753 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/C is initiating minor compaction (all files) 2023-04-21 01:56:15,753 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/C in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:15,753 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/64d25ca4b84d41fdb00981cc29d87ebe, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/c471342b3c4f46a78addb7e03967bd9f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/d3a13ace148e4b9eb477b3aae8a94f13, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/66aeca9ed0b949efa8d36c7e6ec7c7fe] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=47.0 K 2023-04-21 01:56:15,753 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 64d25ca4b84d41fdb00981cc29d87ebe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1682042174467 2023-04-21 01:56:15,754 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting c471342b3c4f46a78addb7e03967bd9f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1682042174806 2023-04-21 01:56:15,754 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting d3a13ace148e4b9eb477b3aae8a94f13, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1682042174930 2023-04-21 01:56:15,754 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 66aeca9ed0b949efa8d36c7e6ec7c7fe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1682042175617 2023-04-21 01:56:15,758 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=209, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/1332d07f86c840f5af8d87d5a354c320 2023-04-21 01:56:15,761 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#C#compaction#566 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:15,769 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:15,770 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:15,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59440 deadline: 1682042235768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:15,773 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:15,774 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/3da8657a573d4f2998277211250ba391 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/3da8657a573d4f2998277211250ba391 2023-04-21 01:56:15,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:15,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59410 deadline: 1682042235771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:15,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682042235773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,778 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/C of 3db499d69ebc07e344a9c8f0976f4e89 into 3da8657a573d4f2998277211250ba391(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:15,778 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:15,778 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/C, priority=12, startTime=1682042175737; duration=0sec 2023-04-21 01:56:15,778 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-21 01:56:15,778 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-21 01:56:15,779 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-21 01:56:15,779 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-21 01:56:15,779 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. because compaction request was cancelled 2023-04-21 01:56:15,779 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-04-21 01:56:15,779 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-21 01:56:15,779 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-21 01:56:15,779 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. because compaction request was cancelled 2023-04-21 01:56:15,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:15,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:15,780 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:15,780 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:15,780 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:15,780 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:15,781 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,781 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:15,781 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/5e834f19a0a044409ef9323f767103d0 2023-04-21 01:56:15,782 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,782 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:15,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:15,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:15,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:15,783 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,783 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:15,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:15,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:15,784 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:15,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:15,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:15,785 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:15,785 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:15,785 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:15,786 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:15,786 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:15,786 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:15,786 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:15,786 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:15,787 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:15,787 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:15,787 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:15,787 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,788 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:15,788 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:15,790 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,790 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:15,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:15,790 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,790 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:15,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:15,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:15,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:15,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:15,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1992affc[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:15,791 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:15,792 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,796 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,796 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:15,796 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:15,796 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:15,796 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,797 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:15,798 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:15,798 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,798 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:15,798 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:15,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:15,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:15,801 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:15,801 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:15,801 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:15,801 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:15,801 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:15,802 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,802 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,802 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:15,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:15,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:15,803 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,804 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:15,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,806 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,806 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:15,806 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:15,806 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:15,806 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:15,806 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:15,806 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:15,806 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,806 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:15,806 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:15,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:15,806 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,807 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,807 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:15,807 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:15,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:15,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:15,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:15,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59440 deadline: 1682042235872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:15,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682042235876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:15,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59410 deadline: 1682042235876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,907 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:15,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:15,909 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:15,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:15,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:15,910 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:15,910 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:15,910 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:15,910 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:15,911 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,911 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:15,913 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:15,913 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:15,913 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,913 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:15,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:15,914 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:15,914 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:15,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:15,914 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:15,914 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:15,914 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:15,914 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:15,914 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:15,915 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:15,915 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:15,915 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:15,915 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:15,915 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:15,915 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:15,915 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,915 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:15,915 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:15,916 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,916 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,916 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,917 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:15,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:15,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:15,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:15,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:15,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:15,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:15,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c6a7785[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:15,918 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:15,918 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,919 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,919 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:15,919 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:15,919 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,920 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:15,919 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:15,921 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,921 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:15,921 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:15,921 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:15,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:15,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:15,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:15,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:15,923 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:15,923 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:15,923 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:15,923 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:15,923 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:15,925 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,925 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:15,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,926 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:15,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,927 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:15,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,928 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:15,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:15,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,930 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,930 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:15,930 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:15,930 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:15,930 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,931 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:15,931 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:15,931 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:15,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:15,931 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:15,931 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:15,931 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:15,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:15,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:15,931 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:15,931 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:15,931 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:15,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:15,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:16,018 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/eae8913ba7db4ef0a06d3656d1e844c6 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/eae8913ba7db4ef0a06d3656d1e844c6 2023-04-21 01:56:16,022 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/A of 3db499d69ebc07e344a9c8f0976f4e89 into eae8913ba7db4ef0a06d3656d1e844c6(size=25.6 K), total size for store is 50.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:16,022 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:16,022 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/A, priority=13, startTime=1682042175553; duration=0sec 2023-04-21 01:56:16,022 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:16,031 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:16,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:16,033 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:16,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:16,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:16,034 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:16,034 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:16,034 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:16,034 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:16,035 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,035 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:16,036 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:16,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:16,036 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,036 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,036 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:16,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:16,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:16,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:16,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:16,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:16,037 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:16,037 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:16,037 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:16,037 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:16,037 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:16,038 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:16,038 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:16,038 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:16,038 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:16,038 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:16,038 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:16,038 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,038 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:16,038 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:16,039 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,039 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:16,040 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,040 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:16,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:16,041 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:16,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69f323a1[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:16,041 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,042 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,042 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:16,042 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:16,042 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,042 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:16,042 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:16,044 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,044 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:16,044 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:16,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:16,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:16,044 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:16,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:16,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:16,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:16,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:16,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:16,047 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:16,047 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:16,047 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:16,047 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:16,047 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:16,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:16,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042236046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,049 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,049 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,049 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:16,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:16,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:16,050 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:16,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,051 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:16,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:16,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042236050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,053 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,053 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:16,053 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:16,053 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:16,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:16,053 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:16,054 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:16,053 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,053 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:16,054 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,054 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:16,054 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:16,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:16,054 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:16,054 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:16,054 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:16,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:16,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:16,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59440 deadline: 1682042236076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:16,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59410 deadline: 1682042236081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:16,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682042236082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,154 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:16,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:16,156 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:16,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:16,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:16,157 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:16,157 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:16,157 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:16,157 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:16,158 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,158 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:16,159 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,159 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:16,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:16,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:16,159 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,159 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:16,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:16,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:16,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:16,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:16,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:16,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:16,160 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:16,160 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:16,160 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:16,160 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:16,160 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:16,160 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:16,161 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:16,161 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:16,161 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:16,161 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:16,161 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:16,161 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,161 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:16,161 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:16,163 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,164 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:16,164 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,164 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:16,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:16,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42698e84[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:16,165 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:16,165 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,166 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,166 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:16,166 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:16,166 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:16,166 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,166 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:16,170 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,170 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:16,170 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:16,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:16,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:16,170 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:16,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:16,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:16,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:16,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:16,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:16,173 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:16,173 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:16,173 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:16,173 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:16,173 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:16,174 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,174 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:16,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,175 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:16,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,176 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:16,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,177 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:16,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,179 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:16,179 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:16,179 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:16,179 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:16,179 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:16,179 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,179 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:16,179 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:16,179 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:16,179 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:16,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:16,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:16,179 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,179 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,179 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,179 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:16,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:16,197 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/a50cd2ca99ec4595960a3447a636de3c 2023-04-21 01:56:16,201 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/1332d07f86c840f5af8d87d5a354c320 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/1332d07f86c840f5af8d87d5a354c320 2023-04-21 01:56:16,204 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/1332d07f86c840f5af8d87d5a354c320, entries=300, sequenceid=209, filesize=45.8 K 2023-04-21 01:56:16,205 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/5e834f19a0a044409ef9323f767103d0 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/5e834f19a0a044409ef9323f767103d0 2023-04-21 01:56:16,208 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/5e834f19a0a044409ef9323f767103d0, entries=150, sequenceid=209, filesize=11.8 K 2023-04-21 01:56:16,209 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/a50cd2ca99ec4595960a3447a636de3c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/a50cd2ca99ec4595960a3447a636de3c 2023-04-21 01:56:16,213 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/a50cd2ca99ec4595960a3447a636de3c, entries=150, sequenceid=209, filesize=11.8 K 2023-04-21 01:56:16,213 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~78.27 KB/80150, heapSize ~218.59 KB/223840, currentSize=100.63 KB/103050 for 3db499d69ebc07e344a9c8f0976f4e89 in 475ms, sequenceid=209, compaction requested=true 2023-04-21 01:56:16,213 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:16,213 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:16,214 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:16,214 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:56:16,214 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:16,214 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:56:16,214 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 98970 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:56:16,215 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/A is initiating minor compaction (all files) 2023-04-21 01:56:16,215 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/A in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:16,215 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/eae8913ba7db4ef0a06d3656d1e844c6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/da6d97a6aebd4a9f94c11ad293da892f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/1332d07f86c840f5af8d87d5a354c320] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=96.7 K 2023-04-21 01:56:16,215 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:56:16,215 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/B is initiating minor compaction (all files) 2023-04-21 01:56:16,215 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/B in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:16,215 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f5e94ad539eb4438ba013adbfacebf35, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/336e9840de8d4df5b5f76cfa03702d7a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/5e834f19a0a044409ef9323f767103d0] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=35.6 K 2023-04-21 01:56:16,215 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting eae8913ba7db4ef0a06d3656d1e844c6, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1682042174930 2023-04-21 01:56:16,216 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting f5e94ad539eb4438ba013adbfacebf35, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1682042174930 2023-04-21 01:56:16,216 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting da6d97a6aebd4a9f94c11ad293da892f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1682042175617 2023-04-21 01:56:16,216 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 336e9840de8d4df5b5f76cfa03702d7a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1682042175617 2023-04-21 01:56:16,216 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 1332d07f86c840f5af8d87d5a354c320, keycount=300, bloomtype=ROW, size=45.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1682042175723 2023-04-21 01:56:16,216 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 5e834f19a0a044409ef9323f767103d0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1682042175728 2023-04-21 01:56:16,222 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#B#compaction#570 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:16,225 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#A#compaction#569 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:16,233 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/3d706b79672c408fba1bf965a759c8d1 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/3d706b79672c408fba1bf965a759c8d1 2023-04-21 01:56:16,237 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/B of 3db499d69ebc07e344a9c8f0976f4e89 into 3d706b79672c408fba1bf965a759c8d1(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:16,238 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:16,238 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/B, priority=13, startTime=1682042176214; duration=0sec 2023-04-21 01:56:16,238 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:16,238 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-21 01:56:16,238 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-21 01:56:16,239 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-21 01:56:16,239 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. because compaction request was cancelled 2023-04-21 01:56:16,279 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:16,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:16,281 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:16,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:16,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:16,283 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:16,283 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:16,283 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:16,283 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:16,283 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,283 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:16,285 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,286 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,286 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:16,286 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:16,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:16,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:16,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:16,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:16,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:16,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:16,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:16,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:16,288 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:16,288 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:16,288 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:16,289 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:16,289 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:16,289 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:16,289 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:16,289 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-21 01:56:16,289 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:16,289 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:16,289 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:16,289 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:16,289 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:16,290 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:16,314 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304216c09abff88fd4606a08e0882f82b87e6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304216c09abff88fd4606a08e0882f82b87e6 2023-04-21 01:56:16,319 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=231, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/fa445f57591b4e73864515e4a84b14be 2023-04-21 01:56:16,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:16,381 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:16,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:16,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682042236393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:16,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59440 deadline: 1682042236395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:16,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59410 deadline: 1682042236396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:16,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682042236497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:16,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59440 deadline: 1682042236500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:16,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59410 deadline: 1682042236500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:16,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682042236702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:16,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59440 deadline: 1682042236704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:16,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59410 deadline: 1682042236704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,729 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/94becbef18394a3b87d0dd540f1f2b96 2023-04-21 01:56:16,738 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/edfcb40378ca40ec89731390c533ec45 2023-04-21 01:56:16,742 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/fa445f57591b4e73864515e4a84b14be as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/fa445f57591b4e73864515e4a84b14be 2023-04-21 01:56:16,745 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/fa445f57591b4e73864515e4a84b14be, entries=150, sequenceid=231, filesize=25.3 K 2023-04-21 01:56:16,746 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/94becbef18394a3b87d0dd540f1f2b96 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/94becbef18394a3b87d0dd540f1f2b96 2023-04-21 01:56:16,750 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/94becbef18394a3b87d0dd540f1f2b96, entries=150, sequenceid=231, filesize=11.8 K 2023-04-21 01:56:16,750 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/edfcb40378ca40ec89731390c533ec45 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/edfcb40378ca40ec89731390c533ec45 2023-04-21 01:56:16,754 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/edfcb40378ca40ec89731390c533ec45, entries=150, sequenceid=231, filesize=11.8 K 2023-04-21 01:56:16,754 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 3db499d69ebc07e344a9c8f0976f4e89 in 465ms, sequenceid=231, compaction requested=true 2023-04-21 01:56:16,754 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:16,754 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:16,755 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:16,755 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:16,755 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,755 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:16,755 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:16,756 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,756 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:16,757 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,757 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:16,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:16,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:16,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:16,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:16,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e08347a[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:16,759 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:16,759 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,760 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,760 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:16,760 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:16,760 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:16,760 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,760 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:16,761 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:16,761 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:16,762 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:16,762 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:16,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:16,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:16,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:16,765 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:16,765 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:16,765 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:16,765 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:16,765 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:16,767 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,767 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:16,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,767 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:16,768 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:16,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,770 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:16,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,773 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,773 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:16,773 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:16,773 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:16,773 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:16,773 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:16,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:16,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:16,773 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:16,773 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:16,773 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:16,773 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:16,774 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,774 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,774 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,774 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:16,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:16,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:16,874 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:16,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:16,876 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:16,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:16,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:16,880 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:16,880 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:16,880 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:16,880 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:16,880 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,880 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:16,881 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:16,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:16,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:16,881 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:16,882 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,882 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:16,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:16,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:16,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:16,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:16,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:16,883 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:16,883 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:16,883 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:16,883 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:16,883 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:16,883 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:16,883 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:16,883 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-21 01:56:16,884 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:16,884 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:16,884 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:16,884 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:16,884 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:16,884 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:16,895 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421bd55f6341e344c059aa19b9e2bacff24 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421bd55f6341e344c059aa19b9e2bacff24 2023-04-21 01:56:16,900 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=246, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/54b9cab515c84816abfb7ec5d7bf2c31 2023-04-21 01:56:16,908 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/f409ecaaa7b34014a59b52b9e7701d95 2023-04-21 01:56:16,933 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/beea4d279260456fbe82de606664c3c7 2023-04-21 01:56:16,938 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/54b9cab515c84816abfb7ec5d7bf2c31 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/54b9cab515c84816abfb7ec5d7bf2c31 2023-04-21 01:56:16,941 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/54b9cab515c84816abfb7ec5d7bf2c31, entries=150, sequenceid=246, filesize=25.3 K 2023-04-21 01:56:16,942 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/f409ecaaa7b34014a59b52b9e7701d95 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f409ecaaa7b34014a59b52b9e7701d95 2023-04-21 01:56:16,946 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f409ecaaa7b34014a59b52b9e7701d95, entries=150, sequenceid=246, filesize=11.8 K 2023-04-21 01:56:16,947 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/beea4d279260456fbe82de606664c3c7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/beea4d279260456fbe82de606664c3c7 2023-04-21 01:56:16,950 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/beea4d279260456fbe82de606664c3c7, entries=150, sequenceid=246, filesize=11.8 K 2023-04-21 01:56:16,951 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 3db499d69ebc07e344a9c8f0976f4e89 in 68ms, sequenceid=246, compaction requested=true 2023-04-21 01:56:16,951 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:16,951 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:16,951 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:16,951 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:16,951 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,951 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:16,951 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:16,953 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,953 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:16,953 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,953 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:16,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:16,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2131b461[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:16,955 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:16,955 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,961 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,961 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:16,961 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:16,961 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,961 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:16,961 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:16,963 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:16,963 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,963 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:16,963 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:16,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:16,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:16,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:16,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:16,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:16,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:16,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:16,965 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:16,966 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:16,966 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:16,966 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:16,966 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:16,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:16,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:16,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:16,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:16,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:16,968 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,969 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:16,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:16,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,971 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,971 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:16,971 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:16,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:16,971 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:16,971 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:16,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:16,972 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:16,971 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:16,971 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:16,972 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:16,972 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:16,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:16,972 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:16,972 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,972 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:16,972 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:16,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:16,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:17,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:17,032 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-21 01:56:17,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:17,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:17,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:17,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:17,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:17,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:17,045 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304212147536e57444d408ec2b309782573cd to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304212147536e57444d408ec2b309782573cd 2023-04-21 01:56:17,059 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/ff6b8c297b864c2195cf7166b031ed79 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/ff6b8c297b864c2195cf7166b031ed79 2023-04-21 01:56:17,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:17,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042237057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:17,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682042177049, max timestamp=1682042177059Num uniques 859; 2023-04-21 01:56:17,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682042177049, max timestamp=1682042177059Num uniques 859; 2023-04-21 01:56:17,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682042177049, max timestamp=1682042177059Num uniques 859; 2023-04-21 01:56:17,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:56:17,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682042237058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:56:17,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:56:17,069 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/A of 3db499d69ebc07e344a9c8f0976f4e89 into ff6b8c297b864c2195cf7166b031ed79(size=25.7 K), total size for store is 76.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:17,069 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:17,069 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/A, priority=13, startTime=1682042176213; duration=0sec 2023-04-21 01:56:17,069 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:17,072 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:17,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:17,074 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:17,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:17,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:17,080 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:17,080 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:17,080 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:17,080 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:17,081 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,081 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:17,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:17,083 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,083 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:17,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042237079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:17,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:17,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:17,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59440 deadline: 1682042237081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,084 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,084 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:17,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:17,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:17,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:17,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:17,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:17,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:17,085 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:17,085 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:17,085 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:17,085 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:17,085 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:17,085 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:17,085 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:17,086 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:17,086 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:17,086 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:17,086 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:17,086 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,086 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:17,086 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:17,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:17,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59410 deadline: 1682042237083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,087 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,087 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:17,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:17,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:17,088 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,088 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:17,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:17,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:17,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:17,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:17,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@512f65cd[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:17,089 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:17,089 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,090 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,090 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:17,090 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:17,090 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,090 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:17,090 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:17,091 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,091 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:17,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:17,091 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:17,092 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:17,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:17,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:17,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:17,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:17,095 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:17,095 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:17,095 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:17,095 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:17,095 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:17,097 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,097 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:17,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,097 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:17,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:17,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:17,098 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:17,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:17,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:17,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:17,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:17,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,099 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:17,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:17,103 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,103 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:17,103 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:17,103 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:17,103 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:17,104 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:17,103 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:17,104 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:17,103 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:17,104 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:17,104 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:17,104 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,104 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,104 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:17,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:17,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:17,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682042237165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:17,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042237185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:17,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59440 deadline: 1682042237185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:17,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59410 deadline: 1682042237187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,204 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:17,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:17,206 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:17,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:17,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:17,208 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:17,208 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:17,208 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:17,208 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:17,208 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,208 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:17,209 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:17,209 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:17,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:17,210 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,210 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:17,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:17,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:17,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:17,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:17,211 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:17,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:17,211 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:17,211 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:17,211 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:17,212 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:17,212 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:17,212 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:17,212 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:17,212 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:17,212 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:17,212 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:17,212 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:17,212 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,212 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:17,212 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:17,215 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,215 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:17,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:17,215 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,215 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:17,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:17,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:17,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:17,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:17,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:17,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fbcdcc2[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:17,217 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:17,217 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,218 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,218 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:17,218 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:17,218 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:17,218 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,218 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:17,219 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:17,219 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:17,219 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,219 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:17,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:17,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:17,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:17,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:17,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:17,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:17,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:17,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:17,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:17,222 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:17,222 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:17,222 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:17,222 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:17,222 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:17,223 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,223 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,223 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:17,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:17,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:17,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:17,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:17,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:17,224 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:17,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:17,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:17,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,225 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:17,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:17,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,228 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,228 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:17,228 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:17,228 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:17,228 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:17,228 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:17,228 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:17,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:17,228 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:17,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:17,228 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:17,228 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,228 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:17,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:17,228 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,228 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,228 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:17,328 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:17,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:17,331 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:17,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:17,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:17,332 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:17,332 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:17,332 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:17,332 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:17,332 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,333 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:17,334 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:17,334 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:17,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:17,334 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,334 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:17,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:17,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:17,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:17,335 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:17,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:17,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:17,336 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:17,336 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:17,336 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:17,336 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:17,336 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:17,336 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:17,336 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:17,336 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:17,336 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:17,336 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:17,336 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:17,336 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,336 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:17,336 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:17,338 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,338 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:17,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:17,338 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,338 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:17,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:17,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:17,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:17,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:17,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:17,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@501888ab[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:17,340 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:17,340 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,341 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,342 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:17,342 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:17,342 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:17,342 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,342 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:17,344 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,344 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:17,344 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:17,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:17,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:17,344 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:17,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:17,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:17,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:17,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:17,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:17,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:17,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:17,347 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:17,347 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:17,347 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:17,347 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:17,347 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:17,349 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,349 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:17,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:17,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,349 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:17,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:17,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:17,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:17,349 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:17,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:17,350 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:17,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:17,352 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,352 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:17,352 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:17,352 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:17,352 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:17,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:17,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:17,352 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:17,353 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:17,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:17,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:17,352 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:17,352 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:17,353 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,353 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,353 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,353 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:17,353 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:17,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:17,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682042237369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:17,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59440 deadline: 1682042237387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:17,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042237388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,394 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:17,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59410 deadline: 1682042237391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,453 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:17,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:17,455 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:17,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:17,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:17,457 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=257, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/6bf9eb90b83c409fac804e8b9ed93be6 2023-04-21 01:56:17,457 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:17,457 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:17,457 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:17,457 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:17,460 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,460 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:17,462 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:17,462 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:17,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:17,462 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,462 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:17,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:17,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:17,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:17,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:17,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:17,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:17,464 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:17,464 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:17,465 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:17,465 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:17,465 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:17,465 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:17,465 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:17,465 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:17,465 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:17,465 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:17,465 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:17,465 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,465 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:17,465 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:17,467 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,467 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:17,468 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,468 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:17,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:17,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ca6bc75[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:17,470 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:17,470 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,471 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/d3da22339d974f71a2f694fc4457135f 2023-04-21 01:56:17,471 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,471 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:17,471 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:17,472 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:17,472 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,472 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:17,473 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:17,473 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:17,473 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:17,473 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:17,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:17,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:17,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:17,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:17,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:17,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:17,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:17,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:17,478 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:17,478 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:17,478 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:17,478 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:17,478 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:17,479 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,479 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,479 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:17,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:17,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:17,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:17,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:17,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:17,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,480 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:17,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:17,482 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/88a7c3afdcc84c969f1ab61e5c2afc13 2023-04-21 01:56:17,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:17,483 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:17,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:17,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,485 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/6bf9eb90b83c409fac804e8b9ed93be6 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/6bf9eb90b83c409fac804e8b9ed93be6 2023-04-21 01:56:17,488 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,488 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:17,489 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:17,489 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,489 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:17,489 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:17,488 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/6bf9eb90b83c409fac804e8b9ed93be6, entries=400, sequenceid=257, filesize=59.5 K 2023-04-21 01:56:17,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:17,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:17,489 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:17,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:17,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:17,489 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:17,489 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:17,489 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:17,489 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:17,490 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:17,489 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:17,490 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,490 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/d3da22339d974f71a2f694fc4457135f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/d3da22339d974f71a2f694fc4457135f 2023-04-21 01:56:17,493 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/d3da22339d974f71a2f694fc4457135f, entries=150, sequenceid=257, filesize=11.8 K 2023-04-21 01:56:17,493 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/88a7c3afdcc84c969f1ab61e5c2afc13 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/88a7c3afdcc84c969f1ab61e5c2afc13 2023-04-21 01:56:17,496 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/88a7c3afdcc84c969f1ab61e5c2afc13, entries=150, sequenceid=257, filesize=11.8 K 2023-04-21 01:56:17,497 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 3db499d69ebc07e344a9c8f0976f4e89 in 465ms, sequenceid=257, compaction requested=true 2023-04-21 01:56:17,497 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:17,497 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:17,497 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:56:17,497 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:56:17,497 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:17,497 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:17,498 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 139002 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:56:17,498 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48604 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:56:17,498 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/A is initiating minor compaction (all files) 2023-04-21 01:56:17,498 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/B is initiating minor compaction (all files) 2023-04-21 01:56:17,498 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/A in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:17,498 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/B in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:17,498 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/ff6b8c297b864c2195cf7166b031ed79, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/fa445f57591b4e73864515e4a84b14be, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/54b9cab515c84816abfb7ec5d7bf2c31, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/6bf9eb90b83c409fac804e8b9ed93be6] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=135.7 K 2023-04-21 01:56:17,498 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/3d706b79672c408fba1bf965a759c8d1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/94becbef18394a3b87d0dd540f1f2b96, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f409ecaaa7b34014a59b52b9e7701d95, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/d3da22339d974f71a2f694fc4457135f] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=47.5 K 2023-04-21 01:56:17,499 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting ff6b8c297b864c2195cf7166b031ed79, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1682042175728 2023-04-21 01:56:17,499 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 3d706b79672c408fba1bf965a759c8d1, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1682042175728 2023-04-21 01:56:17,499 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting fa445f57591b4e73864515e4a84b14be, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682042175762 2023-04-21 01:56:17,499 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 94becbef18394a3b87d0dd540f1f2b96, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682042175762 2023-04-21 01:56:17,500 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 54b9cab515c84816abfb7ec5d7bf2c31, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1682042176392 2023-04-21 01:56:17,500 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting f409ecaaa7b34014a59b52b9e7701d95, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1682042176392 2023-04-21 01:56:17,500 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 6bf9eb90b83c409fac804e8b9ed93be6, keycount=400, bloomtype=ROW, size=59.5 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1682042177019 2023-04-21 01:56:17,500 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting d3da22339d974f71a2f694fc4457135f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1682042177028 2023-04-21 01:56:17,507 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#B#compaction#581 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:17,509 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#A#compaction#580 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:17,521 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/cab2d87a194845a18ecc39c7ec9f99e7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/cab2d87a194845a18ecc39c7ec9f99e7 2023-04-21 01:56:17,525 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/B of 3db499d69ebc07e344a9c8f0976f4e89 into cab2d87a194845a18ecc39c7ec9f99e7(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:17,525 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:17,525 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/B, priority=12, startTime=1682042177497; duration=0sec 2023-04-21 01:56:17,525 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:17,525 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-21 01:56:17,527 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60572 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-21 01:56:17,527 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/C is initiating minor compaction (all files) 2023-04-21 01:56:17,527 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/C in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:17,527 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/3da8657a573d4f2998277211250ba391, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/a50cd2ca99ec4595960a3447a636de3c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/edfcb40378ca40ec89731390c533ec45, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/beea4d279260456fbe82de606664c3c7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/88a7c3afdcc84c969f1ab61e5c2afc13] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=59.2 K 2023-04-21 01:56:17,527 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 3da8657a573d4f2998277211250ba391, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1682042175617 2023-04-21 01:56:17,528 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting a50cd2ca99ec4595960a3447a636de3c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1682042175728 2023-04-21 01:56:17,528 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting edfcb40378ca40ec89731390c533ec45, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682042175762 2023-04-21 01:56:17,528 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting beea4d279260456fbe82de606664c3c7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1682042176392 2023-04-21 01:56:17,528 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 88a7c3afdcc84c969f1ab61e5c2afc13, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1682042177028 2023-04-21 01:56:17,537 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#C#compaction#582 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:17,545 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/4fcf6fc82a494016a0378408609aa54b as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/4fcf6fc82a494016a0378408609aa54b 2023-04-21 01:56:17,549 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/C of 3db499d69ebc07e344a9c8f0976f4e89 into 4fcf6fc82a494016a0378408609aa54b(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:17,549 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:17,549 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/C, priority=11, startTime=1682042177497; duration=0sec 2023-04-21 01:56:17,549 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:17,590 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:17,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:17,592 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:17,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:17,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:17,593 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:17,593 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:17,593 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:17,593 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:17,593 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,593 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:17,595 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:17,595 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:17,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:17,595 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,595 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:17,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:17,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:17,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:17,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:17,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:17,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:17,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:17,596 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:17,596 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:17,596 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:17,597 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:17,597 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:17,597 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:17,597 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:17,597 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-21 01:56:17,597 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:17,597 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:17,597 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:17,597 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:17,597 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:17,597 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:17,614 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421436ed227bb8f4e7581388632334690aa to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421436ed227bb8f4e7581388632334690aa 2023-04-21 01:56:17,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:17,672 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:17,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:17,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042237692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:17,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59440 deadline: 1682042237694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:17,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682042237695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:17,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59410 deadline: 1682042237696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,800 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:17,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59440 deadline: 1682042237798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:17,800 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:17,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682042237798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:18,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59440 deadline: 1682042238001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:18,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682042238002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,020 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=295, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/e2c5f20a9023466bb9746546006a1af0 2023-04-21 01:56:18,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:18,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042238199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:18,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59410 deadline: 1682042238200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:18,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59440 deadline: 1682042238304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:18,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59426 deadline: 1682042238306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,336 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/2451f2f328bc4e04aa20c602824b75d5 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/2451f2f328bc4e04aa20c602824b75d5 2023-04-21 01:56:18,341 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/A of 3db499d69ebc07e344a9c8f0976f4e89 into 2451f2f328bc4e04aa20c602824b75d5(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:18,341 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:18,341 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/A, priority=12, startTime=1682042177497; duration=0sec 2023-04-21 01:56:18,341 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:18,430 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=295 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/e0e875c40ec249c18fc83fe8b92e42c9 2023-04-21 01:56:18,441 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=295 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/eacd4847f1ac49cb9b3ebccea3dc9a02 2023-04-21 01:56:18,445 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/e2c5f20a9023466bb9746546006a1af0 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/e2c5f20a9023466bb9746546006a1af0 2023-04-21 01:56:18,448 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/e2c5f20a9023466bb9746546006a1af0, entries=150, sequenceid=295, filesize=25.4 K 2023-04-21 01:56:18,449 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/e0e875c40ec249c18fc83fe8b92e42c9 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/e0e875c40ec249c18fc83fe8b92e42c9 2023-04-21 01:56:18,452 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/e0e875c40ec249c18fc83fe8b92e42c9, entries=150, sequenceid=295, filesize=11.9 K 2023-04-21 01:56:18,453 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/eacd4847f1ac49cb9b3ebccea3dc9a02 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/eacd4847f1ac49cb9b3ebccea3dc9a02 2023-04-21 01:56:18,457 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/eacd4847f1ac49cb9b3ebccea3dc9a02, entries=150, sequenceid=295, filesize=11.9 K 2023-04-21 01:56:18,457 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 3db499d69ebc07e344a9c8f0976f4e89 in 860ms, sequenceid=295, compaction requested=false 2023-04-21 01:56:18,457 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:18,457 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:18,457 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:18,458 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:18,458 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,458 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:18,458 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:18,459 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:18,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:18,459 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:18,460 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,460 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:18,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:18,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:18,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:18,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@568daf21[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:18,461 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:18,461 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,463 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,463 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:18,463 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:18,463 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:18,464 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,464 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:18,466 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:18,466 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:18,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,466 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:18,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:18,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:18,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:18,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:18,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:18,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:18,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:18,470 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:18,470 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:18,470 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:18,470 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:18,470 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:18,472 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,472 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:18,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,472 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:18,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:18,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:18,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:18,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:18,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,474 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:18,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,475 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:18,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,478 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:18,478 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:18,478 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,478 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:18,478 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:18,478 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,478 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:18,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:18,478 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:18,478 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:18,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:18,478 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,478 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:18,478 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:18,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:18,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:18,478 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,478 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:18,579 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:18,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:18,581 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:18,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:18,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:18,582 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:18,582 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:18,582 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:18,582 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:18,582 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,582 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:18,584 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:18,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:18,584 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:18,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:18,584 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,584 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:18,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:18,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:18,585 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:18,585 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:18,585 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:18,585 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:18,585 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:18,585 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:18,585 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:18,585 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:18,586 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:18,586 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:18,586 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-21 01:56:18,586 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:18,586 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:18,586 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:18,586 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:18,586 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:18,586 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:18,603 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304210808fa8d28a043c89631870be426e623 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304210808fa8d28a043c89631870be426e623 2023-04-21 01:56:18,608 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=307, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/0bd52b7b38c24018bac400d0ea5e7214 2023-04-21 01:56:18,617 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/4a0ee5b83ecd4f5eac86512eeca704ff 2023-04-21 01:56:18,624 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/e2c3440699bb4ac6ba873e5debdb251c 2023-04-21 01:56:18,627 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/0bd52b7b38c24018bac400d0ea5e7214 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/0bd52b7b38c24018bac400d0ea5e7214 2023-04-21 01:56:18,630 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/0bd52b7b38c24018bac400d0ea5e7214, entries=150, sequenceid=307, filesize=25.4 K 2023-04-21 01:56:18,631 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/4a0ee5b83ecd4f5eac86512eeca704ff as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/4a0ee5b83ecd4f5eac86512eeca704ff 2023-04-21 01:56:18,634 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/4a0ee5b83ecd4f5eac86512eeca704ff, entries=150, sequenceid=307, filesize=11.9 K 2023-04-21 01:56:18,635 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/e2c3440699bb4ac6ba873e5debdb251c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/e2c3440699bb4ac6ba873e5debdb251c 2023-04-21 01:56:18,639 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/e2c3440699bb4ac6ba873e5debdb251c, entries=150, sequenceid=307, filesize=11.9 K 2023-04-21 01:56:18,639 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 3db499d69ebc07e344a9c8f0976f4e89 in 53ms, sequenceid=307, compaction requested=true 2023-04-21 01:56:18,639 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:18,639 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:18,639 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:18,639 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:18,640 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,640 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:18,640 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:18,642 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,642 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:18,643 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,643 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:18,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:18,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fee23fe[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:18,645 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:18,645 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,646 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,646 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,646 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:18,646 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:18,646 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:18,646 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:18,647 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,647 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:18,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:18,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:18,647 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:18,648 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:18,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:18,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:18,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:18,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:18,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:18,650 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:18,650 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:18,650 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:18,650 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:18,650 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:18,651 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,651 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,652 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:18,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:18,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:18,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:18,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:18,652 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:18,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,653 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,657 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,657 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:18,657 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:18,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:18,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:18,657 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:18,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:18,657 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:18,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:18,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:18,657 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:18,657 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:18,658 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:18,658 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,658 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,658 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,658 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,658 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:18,658 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:18,758 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:18,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:18,760 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:18,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:18,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:18,762 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:18,762 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:18,762 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:18,762 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:18,762 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,762 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:18,763 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,763 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:18,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:18,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:18,764 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,764 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:18,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:18,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:18,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:18,765 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:18,765 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:18,765 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:18,765 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:18,765 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:18,765 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:18,765 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:18,765 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:18,765 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:18,765 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:18,766 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:18,766 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:18,766 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:18,766 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:18,766 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,766 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:18,766 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:18,767 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,767 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:18,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:18,767 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:18,767 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:18,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:18,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:18,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:18,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ba7b378[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:18,769 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:18,769 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,770 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,770 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:18,770 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:18,770 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:18,770 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,770 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:18,773 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:18,773 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:18,773 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,773 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:18,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:18,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:18,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:18,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:18,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:18,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:18,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:18,776 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:18,776 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:18,776 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:18,776 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:18,776 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:18,777 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,777 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,777 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:18,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:18,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:18,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,778 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:18,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:18,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:18,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:18,779 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:18,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,781 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,781 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:18,781 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:18,781 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:18,781 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:18,781 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,781 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:18,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:18,781 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:18,782 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:18,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:18,782 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:18,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:18,782 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:18,782 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,782 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,782 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:18,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:18,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:18,838 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-21 01:56:18,839 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:18,839 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:18,839 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:18,839 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:18,839 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:18,839 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:18,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042178870, max timestamp=1682042178877Num uniques 859; 2023-04-21 01:56:18,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042178870, max timestamp=1682042178877Num uniques 859; 2023-04-21 01:56:18,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:18,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042178870, max timestamp=1682042178877Num uniques 859; 2023-04-21 01:56:18,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:18,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:18,882 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:18,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:18,884 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:18,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:18,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:18,886 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:18,886 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:18,886 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:18,886 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:18,886 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,886 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:18,892 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:18,892 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:18,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:18,892 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,892 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:18,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:18,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:18,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:18,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:18,893 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:18,893 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:18,894 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:18,894 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:18,894 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:18,894 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:18,894 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:18,894 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:18,894 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:18,894 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:18,894 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:18,894 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:18,894 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:18,894 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,894 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:18,894 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:18,896 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,896 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:18,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:18,896 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,896 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:18,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:18,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:18,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:18,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:18,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@322bfd6d[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:18,897 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:18,897 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,898 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,899 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:18,899 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:18,899 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:18,899 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,899 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:18,900 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,900 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:18,900 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:18,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:18,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:18,900 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:18,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:18,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:18,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:18,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:18,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:18,903 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:18,903 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:18,903 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:18,903 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:18,903 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:18,906 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,906 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:18,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,906 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:18,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:18,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:18,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:18,907 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:18,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:18,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:18,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,908 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:18,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59426 deadline: 1682042238910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:18,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:18,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042238911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,918 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:18,918 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:18,918 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:18,918 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:18,918 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:18,918 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:18,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:18,918 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:18,918 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:18,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:18,918 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:18,918 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:18,918 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:18,919 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,919 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:18,919 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:19,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59426 deadline: 1682042239012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:19,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042239015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,018 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:19,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:19,021 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:19,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:19,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:19,022 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:19,022 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:19,022 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:19,022 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:19,022 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,023 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,024 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,024 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:19,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:19,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:19,024 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,024 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:19,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:19,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,025 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:19,025 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:19,025 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:19,025 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:19,025 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:19,025 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:19,026 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:19,026 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:19,026 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:19,026 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:19,026 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:19,026 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:19,026 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:19,026 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:19,026 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,026 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:19,026 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:19,027 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:19,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:19,028 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:19,028 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,028 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:19,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:19,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:19,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:19,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@964a8ae[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:19,029 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:19,029 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,032 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,032 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:19,032 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:19,032 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:19,032 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,032 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:19,033 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,033 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:19,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:19,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:19,033 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:19,034 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:19,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:19,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:19,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:19,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:19,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:19,037 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:19,037 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:19,037 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:19,037 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:19,037 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:19,039 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,039 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:19,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:19,039 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:19,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,039 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:19,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:19,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:19,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:19,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,041 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,043 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:19,043 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:19,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:19,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:19,043 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,043 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,043 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,043 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,043 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:19,043 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,043 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,043 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,043 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:19,043 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:19,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:19,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:19,044 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:19,044 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:19,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:19,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042239080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,083 DEBUG [Thread-4041] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:56:19,144 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:19,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:19,146 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:19,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:19,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:19,147 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:19,147 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:19,147 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:19,147 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:19,148 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,148 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,150 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:19,150 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:19,150 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,150 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:19,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:19,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:19,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:19,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:19,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:19,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:19,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:19,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:19,152 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:19,152 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:19,152 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:19,152 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:19,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:19,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:19,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:19,152 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:19,154 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,154 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:19,154 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,154 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:19,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:19,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@170b9baf[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:19,156 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:19,156 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,157 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,158 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:19,158 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:19,158 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:19,158 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,158 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:19,159 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,159 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:19,159 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:19,159 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:19,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:19,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:19,163 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:19,163 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:19,163 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:19,163 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:19,163 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:19,165 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,165 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:19,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,166 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:19,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:19,167 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:19,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,168 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:19,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,170 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:19,170 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:19,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:19,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:19,170 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,170 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,170 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,170 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,170 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:19,170 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:19,170 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:19,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:19,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:19,171 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:19,171 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:19,170 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,171 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,171 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:19,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042239208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:19,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59410 deadline: 1682042239213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:19,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59426 deadline: 1682042239213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:19,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042239217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,251 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042182f6a92016f14804a315c7d4e8984542 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042182f6a92016f14804a315c7d4e8984542 2023-04-21 01:56:19,271 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:19,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:19,273 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:19,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:19,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:19,274 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:19,274 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:19,274 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:19,274 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:19,275 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,275 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,276 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,276 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:19,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:19,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:19,276 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,276 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:19,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:19,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:19,277 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:19,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:19,277 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:19,277 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:19,277 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:19,277 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:19,277 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:19,278 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:19,278 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:19,278 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:19,278 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:19,278 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:19,278 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:19,278 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,278 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:19,278 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:19,279 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,279 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:19,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:19,280 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,280 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:19,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:19,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:19,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:19,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:19,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51f1d645[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:19,281 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:19,281 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,285 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,286 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:19,286 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:19,286 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:19,286 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,286 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:19,287 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,287 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:19,287 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:19,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:19,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:19,287 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:19,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:19,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:19,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:19,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:19,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:19,289 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:19,289 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:19,289 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:19,290 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:19,290 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:19,291 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,291 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,291 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:19,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:19,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:19,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:19,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:19,292 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:19,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,293 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:19,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,295 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,295 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:19,295 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:19,295 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,295 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:19,295 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,295 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,295 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,295 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:19,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:19,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:19,295 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:19,296 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:19,295 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:19,296 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:19,296 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:19,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:19,396 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:19,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:19,398 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:19,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:19,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:19,400 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:19,400 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:19,400 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:19,400 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:19,400 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,400 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,402 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:19,402 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:19,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:19,403 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,403 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:19,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:19,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:19,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:19,404 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:19,405 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:19,405 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:19,405 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:19,405 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:19,405 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:19,405 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:19,405 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:19,405 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:19,405 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:19,405 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:19,405 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,405 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:19,405 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:19,407 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:19,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:19,407 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:19,408 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,408 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:19,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:19,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:19,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:19,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c5c9983[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:19,410 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:19,410 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,411 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,411 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:19,411 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:19,411 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:19,411 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,411 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:19,413 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,413 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:19,413 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:19,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:19,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:19,413 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:19,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:19,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:19,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:19,416 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:19,416 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:19,416 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:19,416 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:19,416 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:19,420 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:19,420 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,420 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:19,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:19,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:19,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:19,421 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:19,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,422 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,425 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,425 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,425 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,425 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:19,425 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:19,425 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:19,425 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:19,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:19,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:19,425 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:19,425 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:19,425 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:19,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:19,425 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,426 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,426 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,426 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:19,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:19,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59426 deadline: 1682042239518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:19,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042239518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,526 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:19,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:19,527 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:19,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:19,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:19,528 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:19,529 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:19,529 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:19,529 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:19,529 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,529 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,530 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,530 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:19,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:19,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:19,531 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,531 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:19,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:19,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:19,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:19,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:19,532 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:19,532 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:19,532 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:19,532 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:19,534 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:19,534 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:19,534 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,534 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:19,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:19,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10d1777e[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:19,535 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:19,535 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,538 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,539 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:19,539 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:19,539 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:19,539 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,539 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:19,540 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:19,540 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:19,540 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:19,540 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:19,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:19,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:19,542 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:19,542 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:19,542 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:19,542 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:19,542 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:19,544 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,544 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:19,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:19,545 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,545 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:19,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,547 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:19,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,549 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,549 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,549 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,549 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:19,549 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:19,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:19,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:19,549 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,549 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:19,549 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:19,549 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:19,549 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,549 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,549 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:19,549 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:19,549 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:19,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:19,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:19,649 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:19,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:19,652 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:19,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:19,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:19,653 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:19,653 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:19,653 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:19,653 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:19,654 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,654 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,656 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=318, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/e9d046a803a14dda96d1ca1c7a351814 2023-04-21 01:56:19,656 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,656 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:19,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:19,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:19,657 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,657 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:19,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:19,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:19,659 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:19,659 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:19,659 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:19,659 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:19,659 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:19,659 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:19,659 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:19,659 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:19,660 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:19,660 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:19,660 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:19,660 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:19,661 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:19,661 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,661 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:19,661 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:19,663 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,663 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:19,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:19,663 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,663 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:19,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:19,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:19,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:19,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:19,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b4c4028[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:19,665 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:19,665 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,666 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,666 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/f26fa546fb354ebd931192538dea42b7 2023-04-21 01:56:19,666 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:19,667 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:19,667 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:19,667 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,667 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:19,671 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:19,671 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:19,671 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,671 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:19,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:19,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:19,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:19,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:19,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:19,675 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:19,675 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:19,675 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:19,675 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:19,675 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:19,676 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/8a5c32de03a2431a8ebee09a2950b12c 2023-04-21 01:56:19,677 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,677 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:19,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:19,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,677 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:19,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:19,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:19,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,678 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:19,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:19,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,680 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:19,681 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/e9d046a803a14dda96d1ca1c7a351814 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/e9d046a803a14dda96d1ca1c7a351814 2023-04-21 01:56:19,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,684 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,684 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,684 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,684 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,684 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:19,684 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:19,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:19,684 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,684 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:19,684 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:19,684 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:19,685 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:19,685 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:19,684 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:19,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:19,685 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,685 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/e9d046a803a14dda96d1ca1c7a351814, entries=300, sequenceid=318, filesize=46.1 K 2023-04-21 01:56:19,685 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/f26fa546fb354ebd931192538dea42b7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f26fa546fb354ebd931192538dea42b7 2023-04-21 01:56:19,689 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f26fa546fb354ebd931192538dea42b7, entries=150, sequenceid=318, filesize=11.9 K 2023-04-21 01:56:19,689 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/8a5c32de03a2431a8ebee09a2950b12c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/8a5c32de03a2431a8ebee09a2950b12c 2023-04-21 01:56:19,692 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/8a5c32de03a2431a8ebee09a2950b12c, entries=150, sequenceid=318, filesize=11.9 K 2023-04-21 01:56:19,692 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 3db499d69ebc07e344a9c8f0976f4e89 in 854ms, sequenceid=318, compaction requested=true 2023-04-21 01:56:19,693 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:19,693 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:19,693 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:56:19,693 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:56:19,693 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:19,693 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:19,694 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 125754 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:56:19,694 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49190 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:56:19,694 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/A is initiating minor compaction (all files) 2023-04-21 01:56:19,694 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/B is initiating minor compaction (all files) 2023-04-21 01:56:19,694 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/A in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:19,694 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/B in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:19,694 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/2451f2f328bc4e04aa20c602824b75d5, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/e2c5f20a9023466bb9746546006a1af0, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/0bd52b7b38c24018bac400d0ea5e7214, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/e9d046a803a14dda96d1ca1c7a351814] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=122.8 K 2023-04-21 01:56:19,694 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/cab2d87a194845a18ecc39c7ec9f99e7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/e0e875c40ec249c18fc83fe8b92e42c9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/4a0ee5b83ecd4f5eac86512eeca704ff, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f26fa546fb354ebd931192538dea42b7] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=48.0 K 2023-04-21 01:56:19,695 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 2451f2f328bc4e04aa20c602824b75d5, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1682042177028 2023-04-21 01:56:19,695 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting cab2d87a194845a18ecc39c7ec9f99e7, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1682042177028 2023-04-21 01:56:19,695 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting e2c5f20a9023466bb9746546006a1af0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1682042177075 2023-04-21 01:56:19,695 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting e0e875c40ec249c18fc83fe8b92e42c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1682042177075 2023-04-21 01:56:19,696 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 0bd52b7b38c24018bac400d0ea5e7214, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1682042177676 2023-04-21 01:56:19,696 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 4a0ee5b83ecd4f5eac86512eeca704ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1682042177676 2023-04-21 01:56:19,696 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting e9d046a803a14dda96d1ca1c7a351814, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1682042178823 2023-04-21 01:56:19,696 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting f26fa546fb354ebd931192538dea42b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1682042178829 2023-04-21 01:56:19,703 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#B#compaction#593 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:19,705 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#A#compaction#592 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:19,733 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/64bc65738c5448fba614c10f771d31dc as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/64bc65738c5448fba614c10f771d31dc 2023-04-21 01:56:19,737 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/A of 3db499d69ebc07e344a9c8f0976f4e89 into 64bc65738c5448fba614c10f771d31dc(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:19,737 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:19,737 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/A, priority=12, startTime=1682042179693; duration=0sec 2023-04-21 01:56:19,737 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:19,737 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:56:19,738 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49156 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:56:19,738 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/C is initiating minor compaction (all files) 2023-04-21 01:56:19,738 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/C in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:19,738 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/4fcf6fc82a494016a0378408609aa54b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/eacd4847f1ac49cb9b3ebccea3dc9a02, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/e2c3440699bb4ac6ba873e5debdb251c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/8a5c32de03a2431a8ebee09a2950b12c] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=48.0 K 2023-04-21 01:56:19,739 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 4fcf6fc82a494016a0378408609aa54b, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1682042177028 2023-04-21 01:56:19,739 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting eacd4847f1ac49cb9b3ebccea3dc9a02, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1682042177075 2023-04-21 01:56:19,739 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting e2c3440699bb4ac6ba873e5debdb251c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1682042177676 2023-04-21 01:56:19,740 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 8a5c32de03a2431a8ebee09a2950b12c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1682042178829 2023-04-21 01:56:19,745 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#C#compaction#594 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:19,754 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/518bdabcb6b24ef5b5e91e4400b69d70 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/518bdabcb6b24ef5b5e91e4400b69d70 2023-04-21 01:56:19,759 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/C of 3db499d69ebc07e344a9c8f0976f4e89 into 518bdabcb6b24ef5b5e91e4400b69d70(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:19,759 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:19,759 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/C, priority=12, startTime=1682042179693; duration=0sec 2023-04-21 01:56:19,759 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:19,785 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:19,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:19,787 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:19,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:19,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:19,788 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:19,788 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:19,788 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:19,788 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:19,788 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,788 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,790 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,790 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:19,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:19,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:19,790 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:19,790 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:19,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:19,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:19,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:19,791 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:19,791 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:19,791 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:19,791 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:19,791 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:19,791 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:19,791 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:19,791 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:19,791 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:19,792 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:19,792 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-21 01:56:19,792 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:19,792 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:19,792 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:19,792 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:19,792 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:19,792 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:20,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:20,028 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:20,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:20,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59426 deadline: 1682042240036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:20,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:20,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042240039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:20,121 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/8fe2f804b904495e8a384d75a4b91bb0 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/8fe2f804b904495e8a384d75a4b91bb0 2023-04-21 01:56:20,125 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/B of 3db499d69ebc07e344a9c8f0976f4e89 into 8fe2f804b904495e8a384d75a4b91bb0(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:20,125 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:20,125 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/B, priority=12, startTime=1682042179693; duration=0sec 2023-04-21 01:56:20,125 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:20,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:20,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59426 deadline: 1682042240140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:20,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:20,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042240142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:20,212 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304213546ce539b374bbdb7101e43ce04106f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304213546ce539b374bbdb7101e43ce04106f 2023-04-21 01:56:20,217 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=357, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/ee5ce701517446af9ffad29d3cac5138 2023-04-21 01:56:20,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:20,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59426 deadline: 1682042240343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:20,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:20,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042240346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:20,627 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/c550238701a149ecafb2d633e4c029c9 2023-04-21 01:56:20,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:20,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59426 deadline: 1682042240645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:20,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:20,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042240648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,037 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/cb3db025ce5d4866a241a08d1f4bc071 2023-04-21 01:56:21,044 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/ee5ce701517446af9ffad29d3cac5138 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/ee5ce701517446af9ffad29d3cac5138 2023-04-21 01:56:21,047 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/ee5ce701517446af9ffad29d3cac5138, entries=150, sequenceid=357, filesize=25.4 K 2023-04-21 01:56:21,048 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/c550238701a149ecafb2d633e4c029c9 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/c550238701a149ecafb2d633e4c029c9 2023-04-21 01:56:21,052 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/c550238701a149ecafb2d633e4c029c9, entries=150, sequenceid=357, filesize=11.9 K 2023-04-21 01:56:21,052 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/cb3db025ce5d4866a241a08d1f4bc071 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/cb3db025ce5d4866a241a08d1f4bc071 2023-04-21 01:56:21,056 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/cb3db025ce5d4866a241a08d1f4bc071, entries=150, sequenceid=357, filesize=11.9 K 2023-04-21 01:56:21,056 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 3db499d69ebc07e344a9c8f0976f4e89 in 1264ms, sequenceid=357, compaction requested=false 2023-04-21 01:56:21,056 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:21,056 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:21,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:21,056 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:21,057 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:21,068 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:21,070 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,070 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:21,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:21,071 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,071 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:21,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:21,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:21,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:21,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:21,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@450f6fa[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:21,073 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:21,073 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,074 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,074 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:21,074 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:21,074 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:21,084 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:21,084 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:21,084 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:21,084 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,084 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:21,084 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:21,084 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:21,085 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:21,085 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:21,085 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:21,085 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:21,086 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,086 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:21,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:21,086 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:21,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:21,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:21,087 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,089 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:21,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,092 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:21,092 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:21,092 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:21,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:21,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:21,092 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,092 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,092 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,093 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,093 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,093 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,093 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,092 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:21,092 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:21,093 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:21,093 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:21,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:21,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:21,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:21,152 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-21 01:56:21,153 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:21,153 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:21,153 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:21,153 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:21,153 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:21,153 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:21,186 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:21,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042241184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042181170, max timestamp=1682042181185Num uniques 859; 2023-04-21 01:56:21,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:21,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042181170, max timestamp=1682042181185Num uniques 859; 2023-04-21 01:56:21,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042181170, max timestamp=1682042181185Num uniques 859; 2023-04-21 01:56:21,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:21,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:21,193 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:21,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:21,195 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:21,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:21,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:21,196 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:21,196 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:21,196 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:21,196 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:21,197 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,197 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,199 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:21,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:21,199 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,200 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,200 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:21,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:21,201 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:21,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:21,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:21,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:21,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:21,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:21,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:21,201 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:21,201 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:21,201 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:21,201 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:21,201 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:21,202 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:21,202 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,202 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:21,202 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:21,203 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,203 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:21,203 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,203 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:21,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:21,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c98e772[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:21,205 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:21,205 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,206 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,206 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:21,206 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:21,206 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:21,207 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,207 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:21,208 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,208 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:21,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:21,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:21,208 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:21,208 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:21,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:21,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:21,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:21,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:21,211 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:21,211 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:21,211 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:21,211 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:21,211 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:21,213 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,213 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:21,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:21,214 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:21,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:21,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:21,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:21,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,215 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:21,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,216 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:21,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,219 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:21,219 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:21,219 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,219 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:21,219 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,219 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:21,219 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:21,219 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:21,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:21,219 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,219 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:21,219 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:21,219 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,219 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,219 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:21,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:21,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:21,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59410 deadline: 1682042241226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:21,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042241226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,230 DEBUG [Thread-4049] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:56:21,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:21,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59426 deadline: 1682042241228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:21,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042241288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,319 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:21,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:21,322 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:21,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:21,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:21,323 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:21,323 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:21,323 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:21,323 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:21,323 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,323 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,324 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:21,325 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:21,325 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,325 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:21,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:21,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:21,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:21,326 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:21,326 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:21,326 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:21,326 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:21,326 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:21,326 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:21,327 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:21,327 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:21,327 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:21,327 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:21,327 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:21,327 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,327 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:21,327 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:21,328 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,328 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:21,328 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,329 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:21,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:21,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47690118[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:21,330 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:21,330 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,332 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,332 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:21,332 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:21,332 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:21,333 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,333 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:21,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:21,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042241330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,334 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:21,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:21,334 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:21,334 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:21,334 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:21,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:21,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:21,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59426 deadline: 1682042241333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:21,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:21,336 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:21,336 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:21,336 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:21,337 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:21,337 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:21,338 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,338 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:21,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,338 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:21,339 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:21,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:21,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:21,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,340 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:21,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,346 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,346 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,346 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:21,346 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:21,346 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:21,346 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,346 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,346 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,346 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,346 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,346 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:21,346 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:21,346 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:21,346 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:21,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:21,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:21,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:21,446 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:21,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:21,448 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:21,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:21,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:21,449 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:21,449 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:21,449 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:21,450 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:21,450 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,450 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,451 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:21,451 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,452 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,452 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:21,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:21,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:21,453 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:21,453 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:21,453 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:21,453 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:21,453 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:21,453 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:21,453 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:21,453 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:21,453 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:21,453 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:21,453 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:21,453 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:21,454 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:21,454 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,454 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:21,454 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:21,455 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,455 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:21,455 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,455 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:21,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:21,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e1d54dd[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:21,457 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:21,457 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,458 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,458 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:21,458 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:21,458 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:21,458 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,458 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:21,461 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,461 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:21,461 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:21,461 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:21,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:21,463 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:21,464 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:21,464 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:21,464 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:21,464 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:21,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:21,465 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:21,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:21,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:21,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:21,466 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:21,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,467 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,469 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:21,469 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:21,470 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:21,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:21,470 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:21,470 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:21,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:21,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:21,470 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,470 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,470 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,470 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:21,470 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,470 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:21,470 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,470 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,470 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:21,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:21,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:21,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042241493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:21,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042241534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:21,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59426 deadline: 1682042241537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,564 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421592719bb82b34d8dbf682e3cdb65bf96 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421592719bb82b34d8dbf682e3cdb65bf96 2023-04-21 01:56:21,570 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:21,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:21,572 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:21,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:21,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:21,573 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:21,573 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:21,573 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:21,573 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:21,574 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,574 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,575 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:21,575 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:21,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:21,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,575 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,575 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:21,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:21,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:21,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:21,576 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:21,576 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:21,576 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:21,576 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:21,576 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:21,577 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:21,577 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:21,577 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:21,577 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:21,577 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:21,577 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:21,577 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,577 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:21,577 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:21,579 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,579 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:21,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:21,579 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,579 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:21,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:21,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:21,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:21,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:21,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d798ac2[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:21,581 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:21,581 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,582 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,582 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:21,582 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:21,582 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:21,582 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,582 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:21,583 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:21,583 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:21,583 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:21,584 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:21,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:21,586 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:21,586 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:21,586 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:21,586 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:21,586 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:21,587 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,587 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,587 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:21,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:21,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:21,588 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:21,589 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:21,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,592 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,592 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,592 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,592 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,592 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,592 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,592 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,592 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:21,592 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:21,593 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:21,592 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:21,593 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:21,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:21,593 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:21,593 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:21,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:21,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:21,693 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:21,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:21,695 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:21,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:21,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:21,697 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:21,697 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:21,697 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:21,697 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:21,697 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,697 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,698 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:21,698 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:21,699 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,699 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:21,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:21,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:21,700 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:21,700 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:21,700 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:21,700 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:21,700 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:21,700 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:21,700 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:21,700 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:21,701 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:21,701 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:21,701 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:21,701 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,701 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:21,701 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:21,702 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,702 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:21,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:21,702 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:21,702 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:21,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:21,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66415168[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:21,704 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:21,704 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,705 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:21,706 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,706 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:21,707 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,707 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:21,707 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:21,707 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:21,710 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:21,710 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:21,710 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:21,710 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:21,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:21,710 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:21,712 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,712 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:21,712 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:21,712 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:21,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,714 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,718 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,718 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,718 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,718 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,718 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:21,718 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:21,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:21,718 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,718 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:21,718 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:21,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:21,718 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:21,718 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,719 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,719 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:21,719 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:21,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:21,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:21,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042241798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,819 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:21,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:21,821 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:21,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:21,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:21,822 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:21,823 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:21,823 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:21,823 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:21,823 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,823 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,825 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:21,825 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:21,825 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,825 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:21,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:21,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:21,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:21,826 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:21,826 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:21,826 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:21,827 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:21,827 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:21,827 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:21,827 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:21,827 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:21,827 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:21,827 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:21,827 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:21,827 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,827 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:21,827 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:21,828 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:21,829 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,829 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:21,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:21,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69afed79[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:21,831 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:21,831 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,832 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,832 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:21,832 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:21,832 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:21,832 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,832 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:21,833 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:21,833 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:21,834 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:21,834 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:21,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:21,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:21,837 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:21,837 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:21,837 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:21,837 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:21,837 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:21,840 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,840 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:21,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,840 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:21,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:21,841 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:21,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042241839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:21,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59426 deadline: 1682042241841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,848 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,850 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,850 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,850 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,850 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,850 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:21,850 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:21,850 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:21,850 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:21,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:21,850 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:21,850 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:21,850 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:21,850 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,851 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,850 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:21,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:21,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:21,951 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:21,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:21,953 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:21,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:21,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:21,954 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:21,954 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:21,954 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:21,954 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:21,954 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,954 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,955 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,955 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:21,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:21,956 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,956 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:21,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:21,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:21,957 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:21,957 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:21,957 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:21,957 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:21,957 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:21,957 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:21,957 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:21,957 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:21,957 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:21,957 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:21,957 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:21,957 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,957 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:21,957 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:21,959 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,959 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:21,959 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,959 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:21,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:21,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:21,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d00e72d[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:21,960 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:21,960 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,962 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,962 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:21,962 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:21,962 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:21,962 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,962 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:21,964 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,964 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:21,964 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:21,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:21,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:21,965 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:21,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:21,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:21,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:21,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:21,967 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:21,967 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:21,967 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:21,967 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:21,967 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:21,968 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,968 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:21,968 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:21,969 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=369, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/4ce5468978e143c4aee82a30b92a0db4 2023-04-21 01:56:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,970 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:21,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,971 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,973 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:21,973 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:21,973 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,973 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,973 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:21,973 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:21,973 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:21,973 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:21,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:21,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:21,973 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:21,974 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,974 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:21,974 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:21,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:21,974 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:21,974 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:21,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:21,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:22,074 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:22,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:22,076 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:22,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:22,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:22,078 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:22,078 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:22,078 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:22,078 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:22,078 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,078 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,080 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,080 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:22,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:22,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:22,081 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,081 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:22,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:22,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,082 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:22,082 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:22,082 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:22,082 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:22,082 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:22,082 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:22,083 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:22,083 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:22,083 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:22,083 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:22,083 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:22,083 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:22,083 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:22,083 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:22,083 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,083 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:22,083 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:22,085 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,085 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:22,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:22,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:22,085 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,085 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:22,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:22,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:22,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:22,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c9e5dd5[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:22,087 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:22,087 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,088 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,088 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:22,088 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:22,088 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:22,088 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,089 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:22,090 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,090 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:22,090 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:22,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:22,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:22,090 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:22,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:22,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:22,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:22,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:22,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:22,093 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:22,093 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:22,093 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:22,093 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:22,093 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:22,095 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,095 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:22,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,096 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,096 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:22,098 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:22,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,100 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:22,100 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:22,100 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:22,100 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:22,100 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:22,100 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:22,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:22,100 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,100 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,100 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:22,101 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:22,100 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,101 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,101 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,101 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:22,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:22,201 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:22,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:22,203 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:22,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:22,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:22,204 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:22,204 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:22,204 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:22,204 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:22,205 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,205 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,208 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,208 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:22,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:22,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:22,209 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,209 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:22,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:22,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:22,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:22,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:22,210 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:22,210 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:22,210 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:22,210 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:22,210 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:22,211 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:22,211 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:22,211 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:22,211 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:22,211 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:22,211 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:22,211 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,211 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:22,211 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:22,213 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,213 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:22,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:22,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:22,214 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,214 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:22,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:22,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:22,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:22,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20b89f3f[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:22,216 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:22,216 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,217 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,217 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:22,217 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:22,217 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:22,217 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,217 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:22,219 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,219 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:22,219 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:22,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:22,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:22,219 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:22,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:22,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:22,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:22,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:22,222 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:22,222 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:22,222 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:22,222 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:22,222 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:22,223 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,223 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:22,223 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:22,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:22,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:22,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:22,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:22,225 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:22,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,226 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:22,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,231 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,231 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:22,231 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:22,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:22,231 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:22,231 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,231 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,231 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,231 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:22,231 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:22,232 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:22,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:22,232 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,231 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:22,232 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:22,232 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:22,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:22,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042242301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,332 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:22,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:22,334 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:22,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:22,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:22,335 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:22,335 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:22,336 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:22,336 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:22,336 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,336 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,337 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:22,337 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:22,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:22,337 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,338 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:22,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:22,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:22,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:22,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:22,338 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:22,339 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:22,339 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:22,339 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:22,339 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:22,339 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:22,339 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:22,339 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:22,339 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:22,339 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:22,339 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:22,339 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,339 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:22,339 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:22,340 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,340 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:22,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:22,341 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,341 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:22,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:22,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:22,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:22,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:22,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@366a7a74[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:22,343 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:22,343 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,344 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,344 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:22,344 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:22,344 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:22,345 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,345 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:22,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:22,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59426 deadline: 1682042242344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,347 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,347 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:22,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:22,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:22,347 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:22,347 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:22,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:22,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:22,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042242345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:22,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:22,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:22,349 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:22,350 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:22,350 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:22,350 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:22,350 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:22,351 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,351 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:22,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:22,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,351 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:22,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,352 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,353 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:22,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,355 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,355 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:22,355 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:22,355 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:22,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:22,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:22,355 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,355 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:22,355 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:22,356 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:22,356 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,356 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,356 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,356 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,356 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:22,356 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:22,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:22,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:22,379 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=369 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/c210c198f562487dbaf2cf710e01f118 2023-04-21 01:56:22,456 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:22,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:22,458 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:22,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:22,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:22,459 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:22,460 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:22,460 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:22,460 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:22,460 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,460 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,461 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,461 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:22,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:22,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:22,462 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,462 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:22,463 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:22,463 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:22,463 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:22,463 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:22,463 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:22,463 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:22,463 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:22,463 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:22,463 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:22,463 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:22,464 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:22,464 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:22,464 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:22,464 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,464 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:22,464 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:22,466 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,466 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:22,466 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,467 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:22,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:22,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c29ca1b[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:22,468 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:22,468 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,469 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,469 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:22,470 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:22,470 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:22,470 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,470 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:22,471 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:22,471 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,471 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:22,471 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:22,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:22,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:22,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:22,474 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:22,474 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:22,474 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:22,474 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:22,474 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:22,475 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,475 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:22,475 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:22,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:22,476 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,477 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:22,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,480 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,480 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,480 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,480 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,481 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,481 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:22,481 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:22,481 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:22,481 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:22,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:22,481 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:22,482 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:22,481 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:22,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:22,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:22,481 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,484 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:22,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:22,581 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:22,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:22,583 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:22,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:22,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:22,585 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:22,585 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:22,585 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:22,585 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:22,585 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,585 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,586 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:22,586 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:22,587 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,587 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:22,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:22,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:22,588 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:22,588 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:22,588 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:22,588 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:22,588 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:22,589 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:22,589 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:22,589 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:22,589 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:22,589 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:22,589 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:22,589 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,589 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:22,589 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:22,593 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,593 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:22,593 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,593 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:22,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:22,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:22,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:22,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@726746be[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:22,595 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:22,595 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,596 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,596 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:22,596 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:22,596 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:22,597 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,597 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:22,598 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,598 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:22,598 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:22,598 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:22,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:22,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:22,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:22,600 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:22,600 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:22,600 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:22,600 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:22,600 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:22,603 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,603 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:22,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:22,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,603 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:22,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:22,604 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:22,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,605 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:22,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,607 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:22,607 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:22,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:22,608 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:22,607 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,608 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:22,608 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,608 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,608 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,608 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,608 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,608 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:22,608 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:22,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:22,608 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:22,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:22,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:22,608 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:22,708 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:22,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:22,710 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:22,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:22,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:22,712 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:22,712 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:22,712 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:22,712 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:22,712 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,712 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,713 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,713 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:22,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:22,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:22,713 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,714 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:22,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:22,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:22,714 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:22,715 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:22,715 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:22,715 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:22,715 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:22,715 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:22,715 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:22,715 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:22,715 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:22,715 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:22,715 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:22,715 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,716 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:22,716 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:22,717 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,717 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:22,717 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,718 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:22,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:22,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:22,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16c26eaa[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:22,718 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:22,718 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,720 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:22,721 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,721 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:22,722 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:22,722 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:22,722 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,722 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:22,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:22,724 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:22,724 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:22,724 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:22,724 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:22,724 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:22,728 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,728 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:22,728 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:22,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:22,729 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:22,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,730 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:22,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:22,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,732 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:22,732 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:22,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:22,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:22,732 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:22,732 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:22,732 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,732 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:22,732 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:22,732 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,732 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,732 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,732 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:22,732 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:22,732 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:22,732 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:22,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:22,788 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=369 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/12a8f91ba9ac4edbb0dabbfa4d3b0a8e 2023-04-21 01:56:22,794 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/4ce5468978e143c4aee82a30b92a0db4 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/4ce5468978e143c4aee82a30b92a0db4 2023-04-21 01:56:22,797 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/4ce5468978e143c4aee82a30b92a0db4, entries=200, sequenceid=369, filesize=32.3 K 2023-04-21 01:56:22,798 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/c210c198f562487dbaf2cf710e01f118 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/c210c198f562487dbaf2cf710e01f118 2023-04-21 01:56:22,801 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/c210c198f562487dbaf2cf710e01f118, entries=150, sequenceid=369, filesize=11.9 K 2023-04-21 01:56:22,802 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/12a8f91ba9ac4edbb0dabbfa4d3b0a8e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/12a8f91ba9ac4edbb0dabbfa4d3b0a8e 2023-04-21 01:56:22,805 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/12a8f91ba9ac4edbb0dabbfa4d3b0a8e, entries=150, sequenceid=369, filesize=11.9 K 2023-04-21 01:56:22,806 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 3db499d69ebc07e344a9c8f0976f4e89 in 1653ms, sequenceid=369, compaction requested=true 2023-04-21 01:56:22,806 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:22,806 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:22,806 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:56:22,806 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:56:22,806 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:22,806 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:22,807 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85860 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:56:22,807 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/A is initiating minor compaction (all files) 2023-04-21 01:56:22,807 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/A in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:22,807 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/64bc65738c5448fba614c10f771d31dc, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/ee5ce701517446af9ffad29d3cac5138, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/4ce5468978e143c4aee82a30b92a0db4] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=83.8 K 2023-04-21 01:56:22,807 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:56:22,808 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/B is initiating minor compaction (all files) 2023-04-21 01:56:22,808 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 64bc65738c5448fba614c10f771d31dc, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1682042178829 2023-04-21 01:56:22,808 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/B in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:22,808 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/8fe2f804b904495e8a384d75a4b91bb0, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/c550238701a149ecafb2d633e4c029c9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/c210c198f562487dbaf2cf710e01f118] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=36.4 K 2023-04-21 01:56:22,808 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting ee5ce701517446af9ffad29d3cac5138, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1682042178901 2023-04-21 01:56:22,808 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 8fe2f804b904495e8a384d75a4b91bb0, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1682042178829 2023-04-21 01:56:22,808 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 4ce5468978e143c4aee82a30b92a0db4, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1682042180035 2023-04-21 01:56:22,809 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting c550238701a149ecafb2d633e4c029c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1682042178901 2023-04-21 01:56:22,809 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting c210c198f562487dbaf2cf710e01f118, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1682042180035 2023-04-21 01:56:22,815 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#B#compaction#602 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:22,817 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#A#compaction#601 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:22,832 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:22,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:22,835 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:22,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:22,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:22,836 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:22,836 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:22,836 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:22,836 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:22,837 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,837 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:22,839 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:22,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:22,840 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:22,840 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:22,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:22,841 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:22,841 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:22,841 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:22,841 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:22,841 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:22,841 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:22,844 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:22,844 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:22,844 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:22,844 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:22,844 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-21 01:56:22,844 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:22,844 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:22,845 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:22,845 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:22,845 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:22,845 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:23,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:23,116 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:23,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:23,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042243135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:23,226 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/b886f51645c64088985207d4b2b4de47 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b886f51645c64088985207d4b2b4de47 2023-04-21 01:56:23,230 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/B of 3db499d69ebc07e344a9c8f0976f4e89 into b886f51645c64088985207d4b2b4de47(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:23,230 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:23,230 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/B, priority=13, startTime=1682042182806; duration=0sec 2023-04-21 01:56:23,230 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:23,230 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:56:23,232 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:56:23,232 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/C is initiating minor compaction (all files) 2023-04-21 01:56:23,232 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/C in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:23,232 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/518bdabcb6b24ef5b5e91e4400b69d70, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/cb3db025ce5d4866a241a08d1f4bc071, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/12a8f91ba9ac4edbb0dabbfa4d3b0a8e] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=36.3 K 2023-04-21 01:56:23,233 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 518bdabcb6b24ef5b5e91e4400b69d70, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1682042178829 2023-04-21 01:56:23,234 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting cb3db025ce5d4866a241a08d1f4bc071, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1682042178901 2023-04-21 01:56:23,234 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 12a8f91ba9ac4edbb0dabbfa4d3b0a8e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1682042180035 2023-04-21 01:56:23,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:23,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042243238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:23,253 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#C#compaction#604 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:23,262 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304211e8531ca41a5475db41c7323f681fe80 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304211e8531ca41a5475db41c7323f681fe80 2023-04-21 01:56:23,262 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/831cf1cd13d346139c347af578161076 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/831cf1cd13d346139c347af578161076 2023-04-21 01:56:23,267 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=405, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/bc865a4c5cde448ba3293d74ca313086 2023-04-21 01:56:23,267 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/C of 3db499d69ebc07e344a9c8f0976f4e89 into 831cf1cd13d346139c347af578161076(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:23,267 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:23,267 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/C, priority=13, startTime=1682042182806; duration=0sec 2023-04-21 01:56:23,268 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:23,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:23,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042243312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:23,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:23,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042243349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:23,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:23,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59426 deadline: 1682042243352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:23,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:23,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042243441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:23,654 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/b6ad7c5a5c804e4ca1b79140936b9d81 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/b6ad7c5a5c804e4ca1b79140936b9d81 2023-04-21 01:56:23,658 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/A of 3db499d69ebc07e344a9c8f0976f4e89 into b6ad7c5a5c804e4ca1b79140936b9d81(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:23,659 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:23,659 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/A, priority=13, startTime=1682042182806; duration=0sec 2023-04-21 01:56:23,659 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:23,678 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/dc70ef6de0364f5c942af3818c0feac1 2023-04-21 01:56:23,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:23,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042243744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,088 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/613effdeb7e64249a433e37d58011ce9 2023-04-21 01:56:24,091 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/bc865a4c5cde448ba3293d74ca313086 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/bc865a4c5cde448ba3293d74ca313086 2023-04-21 01:56:24,095 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/bc865a4c5cde448ba3293d74ca313086, entries=150, sequenceid=405, filesize=25.4 K 2023-04-21 01:56:24,095 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/dc70ef6de0364f5c942af3818c0feac1 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/dc70ef6de0364f5c942af3818c0feac1 2023-04-21 01:56:24,099 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/dc70ef6de0364f5c942af3818c0feac1, entries=150, sequenceid=405, filesize=11.9 K 2023-04-21 01:56:24,099 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/613effdeb7e64249a433e37d58011ce9 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/613effdeb7e64249a433e37d58011ce9 2023-04-21 01:56:24,102 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/613effdeb7e64249a433e37d58011ce9, entries=150, sequenceid=405, filesize=11.9 K 2023-04-21 01:56:24,103 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 3db499d69ebc07e344a9c8f0976f4e89 in 1259ms, sequenceid=405, compaction requested=false 2023-04-21 01:56:24,103 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:24,103 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:24,104 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,105 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:24,105 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:24,105 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:24,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:24,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@731e1fdc[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:24,107 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:24,107 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:24,108 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:24,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:24,108 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:24,108 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:24,108 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:24,108 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,108 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:24,109 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:24,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:24,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:24,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:24,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:24,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:24,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:24,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:24,112 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:24,112 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:24,112 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:24,112 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:24,112 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:24,114 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:24,114 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:24,114 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:24,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:24,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:24,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:24,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:24,115 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:24,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:24,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:24,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:24,116 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:24,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:24,118 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,118 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:24,118 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:24,118 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,118 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:24,118 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:24,118 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:24,118 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:24,118 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:24,118 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:24,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:24,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:24,118 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:24,118 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:24,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:24,119 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:24,119 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:24,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:24,219 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:24,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:24,221 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:24,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:24,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:24,222 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:24,222 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:24,222 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:24,222 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:24,223 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:24,223 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:24,224 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:24,224 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:24,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:24,225 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,225 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:24,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:24,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:24,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:24,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:24,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:24,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:24,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:24,226 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:24,226 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:24,226 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:24,226 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:24,226 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:24,227 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:24,227 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:24,227 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-21 01:56:24,227 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:24,227 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:24,227 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:24,227 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:24,227 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:24,227 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:24,240 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421cc7097028ba646739f032a229a253257 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421cc7097028ba646739f032a229a253257 2023-04-21 01:56:24,249 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=419, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/e23bf632e3aa4c898586f4e7f713eba9 2023-04-21 01:56:24,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:24,251 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:24,261 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/8763dbe684b74fdba80bb8dbfc6e07b6 2023-04-21 01:56:24,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042184286, max timestamp=1682042184304Num uniques 859; 2023-04-21 01:56:24,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:24,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042184286, max timestamp=1682042184304Num uniques 859; 2023-04-21 01:56:24,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042184286, max timestamp=1682042184304Num uniques 859; 2023-04-21 01:56:24,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:24,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:24,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:24,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042244353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:24,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042244457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:24,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59448 deadline: 1682042244660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,674 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/653c05bf8ec34e6db7d1c5a4a76be14a 2023-04-21 01:56:24,678 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/e23bf632e3aa4c898586f4e7f713eba9 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/e23bf632e3aa4c898586f4e7f713eba9 2023-04-21 01:56:24,681 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/e23bf632e3aa4c898586f4e7f713eba9, entries=150, sequenceid=419, filesize=25.4 K 2023-04-21 01:56:24,682 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/8763dbe684b74fdba80bb8dbfc6e07b6 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/8763dbe684b74fdba80bb8dbfc6e07b6 2023-04-21 01:56:24,685 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/8763dbe684b74fdba80bb8dbfc6e07b6, entries=150, sequenceid=419, filesize=11.9 K 2023-04-21 01:56:24,686 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/653c05bf8ec34e6db7d1c5a4a76be14a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/653c05bf8ec34e6db7d1c5a4a76be14a 2023-04-21 01:56:24,689 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/653c05bf8ec34e6db7d1c5a4a76be14a, entries=150, sequenceid=419, filesize=11.9 K 2023-04-21 01:56:24,689 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 3db499d69ebc07e344a9c8f0976f4e89 in 462ms, sequenceid=419, compaction requested=true 2023-04-21 01:56:24,689 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:24,689 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:24,689 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:24,690 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:24,690 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,690 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:24,690 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:24,691 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:24,691 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:24,691 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:24,691 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:24,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:24,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:24,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:24,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:24,693 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:24,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71656ab6[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:24,693 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:24,695 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:24,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:24,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:24,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:24,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:24,695 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:24,696 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,696 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:24,705 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:24,705 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:24,705 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,705 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:24,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:24,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:24,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:24,707 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:24,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:24,707 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:24,708 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:24,708 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:24,708 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:24,709 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:24,709 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:24,709 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:24,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:24,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:24,710 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:24,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:24,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:24,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:24,711 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:24,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:24,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,714 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,714 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:24,714 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:24,714 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:24,714 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:24,714 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,714 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:24,714 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:24,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:24,714 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:24,714 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:24,714 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:24,714 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:24,714 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:24,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:24,714 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:24,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:24,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:24,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:24,815 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:24,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:24,817 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:24,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:24,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:24,818 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:24,818 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:24,818 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:24,818 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:24,819 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:24,819 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:24,821 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:24,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:24,821 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:24,821 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:24,821 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:24,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:24,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:24,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:24,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:24,822 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:24,822 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:24,822 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:24,822 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:24,822 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:24,822 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:24,822 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:24,822 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-21 01:56:24,823 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:24,823 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:24,823 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:24,823 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:24,823 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:24,823 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:24,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:24,965 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:24,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:24,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042244993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:25,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042245099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,236 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:25,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59410 deadline: 1682042245235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,241 DEBUG [Thread-4049] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:56:25,242 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304210093fe56cdf04f1d85ea3ffa60c403ea to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304210093fe56cdf04f1d85ea3ffa60c403ea 2023-04-21 01:56:25,247 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=455, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/aa48e7582b934dc4a1b9996c45f07196 2023-04-21 01:56:25,256 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/8d88c0739b66420e862def9da551c0e2 2023-04-21 01:56:25,280 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/c513bf031c8944778da2938f71f4fbd1 2023-04-21 01:56:25,284 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/aa48e7582b934dc4a1b9996c45f07196 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/aa48e7582b934dc4a1b9996c45f07196 2023-04-21 01:56:25,288 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/aa48e7582b934dc4a1b9996c45f07196, entries=150, sequenceid=455, filesize=25.4 K 2023-04-21 01:56:25,289 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/8d88c0739b66420e862def9da551c0e2 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/8d88c0739b66420e862def9da551c0e2 2023-04-21 01:56:25,292 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/8d88c0739b66420e862def9da551c0e2, entries=150, sequenceid=455, filesize=11.9 K 2023-04-21 01:56:25,292 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/c513bf031c8944778da2938f71f4fbd1 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/c513bf031c8944778da2938f71f4fbd1 2023-04-21 01:56:25,297 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/c513bf031c8944778da2938f71f4fbd1, entries=150, sequenceid=455, filesize=11.9 K 2023-04-21 01:56:25,297 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 3db499d69ebc07e344a9c8f0976f4e89 in 475ms, sequenceid=455, compaction requested=true 2023-04-21 01:56:25,297 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:25,297 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:25,298 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:25,298 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:25,298 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,298 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:25,298 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:25,299 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,299 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:25,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:25,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:25,299 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:25,300 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:25,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:25,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:25,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:25,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:25,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:25,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@397f9092[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:25,302 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:25,302 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:25,303 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:25,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:25,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:25,303 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:25,303 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:25,303 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:25,303 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,303 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:25,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:25,304 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-21 01:56:25,304 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:25,304 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:25,304 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:25,304 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:25,304 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:25,304 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:25,304 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:25,304 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:25,304 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:25,304 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:25,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:25,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:25,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:25,309 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:25,309 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:25,309 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:25,309 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:25,309 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:25,312 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:25,312 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:25,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:25,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:25,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:25,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:25,313 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:25,313 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:25,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:25,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:25,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:25,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:25,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:25,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:25,315 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:25,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:25,317 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:25,317 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:25,317 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421d19c37ab4b9a4a2384bd6187e146f196 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421d19c37ab4b9a4a2384bd6187e146f196 2023-04-21 01:56:25,317 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,318 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:25,318 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:25,318 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,318 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:25,318 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:25,318 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:25,318 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:25,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:25,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:25,318 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:25,318 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:25,318 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:25,318 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:25,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:25,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:25,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:25,323 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=467, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/d3129a8a9c9a4970a2b771b3c64b1d03 2023-04-21 01:56:25,332 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/63a91fa2e82242f3a18e9017e736f0d6 2023-04-21 01:56:25,356 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/eafaeda6fb07432ebd476c0d066f2ec9 2023-04-21 01:56:25,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:25,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042245356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:25,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042245357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:25,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042245358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,361 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/d3129a8a9c9a4970a2b771b3c64b1d03 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/d3129a8a9c9a4970a2b771b3c64b1d03 2023-04-21 01:56:25,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:25,365 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/d3129a8a9c9a4970a2b771b3c64b1d03, entries=200, sequenceid=467, filesize=32.3 K 2023-04-21 01:56:25,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59426 deadline: 1682042245363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,367 DEBUG [Thread-4045] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:56:25,367 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/63a91fa2e82242f3a18e9017e736f0d6 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/63a91fa2e82242f3a18e9017e736f0d6 2023-04-21 01:56:25,371 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/63a91fa2e82242f3a18e9017e736f0d6, entries=150, sequenceid=467, filesize=11.9 K 2023-04-21 01:56:25,371 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/eafaeda6fb07432ebd476c0d066f2ec9 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/eafaeda6fb07432ebd476c0d066f2ec9 2023-04-21 01:56:25,375 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/eafaeda6fb07432ebd476c0d066f2ec9, entries=150, sequenceid=467, filesize=11.9 K 2023-04-21 01:56:25,376 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 3db499d69ebc07e344a9c8f0976f4e89 in 71ms, sequenceid=467, compaction requested=true 2023-04-21 01:56:25,376 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:25,376 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:25,376 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-21 01:56:25,376 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-21 01:56:25,376 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:25,376 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:25,377 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 138054 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-21 01:56:25,377 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/A is initiating minor compaction (all files) 2023-04-21 01:56:25,377 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/A in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:25,377 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/b6ad7c5a5c804e4ca1b79140936b9d81, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/bc865a4c5cde448ba3293d74ca313086, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/e23bf632e3aa4c898586f4e7f713eba9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/aa48e7582b934dc4a1b9996c45f07196, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/d3129a8a9c9a4970a2b771b3c64b1d03] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=134.8 K 2023-04-21 01:56:25,378 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61714 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-21 01:56:25,378 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/B is initiating minor compaction (all files) 2023-04-21 01:56:25,378 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/B in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:25,378 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b886f51645c64088985207d4b2b4de47, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/dc70ef6de0364f5c942af3818c0feac1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/8763dbe684b74fdba80bb8dbfc6e07b6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/8d88c0739b66420e862def9da551c0e2, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/63a91fa2e82242f3a18e9017e736f0d6] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=60.3 K 2023-04-21 01:56:25,379 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting b886f51645c64088985207d4b2b4de47, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1682042180035 2023-04-21 01:56:25,379 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting b6ad7c5a5c804e4ca1b79140936b9d81, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1682042180035 2023-04-21 01:56:25,379 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting dc70ef6de0364f5c942af3818c0feac1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1682042181221 2023-04-21 01:56:25,380 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting bc865a4c5cde448ba3293d74ca313086, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1682042181221 2023-04-21 01:56:25,380 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 8763dbe684b74fdba80bb8dbfc6e07b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1682042183124 2023-04-21 01:56:25,380 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting e23bf632e3aa4c898586f4e7f713eba9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1682042183124 2023-04-21 01:56:25,380 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 8d88c0739b66420e862def9da551c0e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1682042184331 2023-04-21 01:56:25,380 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting aa48e7582b934dc4a1b9996c45f07196, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1682042184331 2023-04-21 01:56:25,381 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 63a91fa2e82242f3a18e9017e736f0d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1682042184984 2023-04-21 01:56:25,381 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting d3129a8a9c9a4970a2b771b3c64b1d03, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1682042184984 2023-04-21 01:56:25,389 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#B#compaction#616 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:25,418 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:25,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:25,419 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#A#compaction#617 average throughput is 0.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:25,421 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:25,424 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/b75a9dece28c458b93e00081b0f7d080 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b75a9dece28c458b93e00081b0f7d080 2023-04-21 01:56:25,428 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/B of 3db499d69ebc07e344a9c8f0976f4e89 into b75a9dece28c458b93e00081b0f7d080(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:25,428 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:25,428 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/B, priority=11, startTime=1682042185376; duration=0sec 2023-04-21 01:56:25,428 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:25,428 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-21 01:56:25,429 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-21 01:56:25,429 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/C is initiating minor compaction (all files) 2023-04-21 01:56:25,429 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/C in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:25,429 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/831cf1cd13d346139c347af578161076, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/613effdeb7e64249a433e37d58011ce9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/653c05bf8ec34e6db7d1c5a4a76be14a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/c513bf031c8944778da2938f71f4fbd1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/eafaeda6fb07432ebd476c0d066f2ec9] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=60.2 K 2023-04-21 01:56:25,430 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 831cf1cd13d346139c347af578161076, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1682042180035 2023-04-21 01:56:25,430 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 613effdeb7e64249a433e37d58011ce9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1682042181221 2023-04-21 01:56:25,430 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 653c05bf8ec34e6db7d1c5a4a76be14a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1682042183124 2023-04-21 01:56:25,431 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting c513bf031c8944778da2938f71f4fbd1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1682042184331 2023-04-21 01:56:25,431 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting eafaeda6fb07432ebd476c0d066f2ec9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1682042184984 2023-04-21 01:56:25,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:25,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:25,432 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:25,432 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:25,432 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:25,432 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:25,433 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:25,433 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:25,434 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:25,434 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:25,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:25,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:25,434 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,435 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:25,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:25,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:25,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:25,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:25,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:25,444 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:25,444 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:25,444 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:25,446 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#C#compaction#618 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:25,449 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:25,449 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:25,449 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:25,449 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:25,450 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-21 01:56:25,450 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:25,450 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:25,450 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:25,450 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:25,450 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:25,450 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:25,461 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/9083e003a7fd4ba2ae67fc6969575cb7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/9083e003a7fd4ba2ae67fc6969575cb7 2023-04-21 01:56:25,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:25,465 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:25,467 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/C of 3db499d69ebc07e344a9c8f0976f4e89 into 9083e003a7fd4ba2ae67fc6969575cb7(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:25,467 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:25,467 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/C, priority=11, startTime=1682042185376; duration=0sec 2023-04-21 01:56:25,467 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:25,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:25,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042245478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:25,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042245479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,483 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:25,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042245480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,485 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421b6b5a29b282343f7b986e2c3f4c1cd9b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421b6b5a29b282343f7b986e2c3f4c1cd9b 2023-04-21 01:56:25,495 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=488, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/65bcdff729d74de882c7f6d2015f44bd 2023-04-21 01:56:25,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:25,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:25,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042245583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042245583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:25,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042245584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:25,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042245786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:25,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:25,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042245786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042245786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:25,851 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/fd7040044eff43f3a7007256de9a9964 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/fd7040044eff43f3a7007256de9a9964 2023-04-21 01:56:25,855 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/A of 3db499d69ebc07e344a9c8f0976f4e89 into fd7040044eff43f3a7007256de9a9964(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:25,855 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:25,855 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/A, priority=11, startTime=1682042185376; duration=0sec 2023-04-21 01:56:25,855 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:25,910 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/07004024efd44692bcacc62889e20de3 2023-04-21 01:56:26,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:26,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042246090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:26,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042246091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:26,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042246092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,320 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/d96b4358870f4606bcd24e350bdc4e52 2023-04-21 01:56:26,324 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/65bcdff729d74de882c7f6d2015f44bd as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/65bcdff729d74de882c7f6d2015f44bd 2023-04-21 01:56:26,327 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/65bcdff729d74de882c7f6d2015f44bd, entries=150, sequenceid=488, filesize=25.4 K 2023-04-21 01:56:26,328 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/07004024efd44692bcacc62889e20de3 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/07004024efd44692bcacc62889e20de3 2023-04-21 01:56:26,333 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/07004024efd44692bcacc62889e20de3, entries=150, sequenceid=488, filesize=11.9 K 2023-04-21 01:56:26,334 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/d96b4358870f4606bcd24e350bdc4e52 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/d96b4358870f4606bcd24e350bdc4e52 2023-04-21 01:56:26,337 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/d96b4358870f4606bcd24e350bdc4e52, entries=150, sequenceid=488, filesize=11.9 K 2023-04-21 01:56:26,337 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 3db499d69ebc07e344a9c8f0976f4e89 in 887ms, sequenceid=488, compaction requested=false 2023-04-21 01:56:26,337 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:26,337 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:26,337 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:26,338 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:26,338 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,338 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:26,338 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:26,339 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:26,339 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:26,339 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:26,340 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:26,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:26,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:26,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:26,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:26,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:26,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:26,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c4d0e29[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:26,342 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:26,342 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:26,344 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:26,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:26,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:26,344 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:26,344 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,344 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:26,344 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:26,344 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:26,346 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,346 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:26,346 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:26,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:26,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:26,346 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:26,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:26,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:26,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:26,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:26,349 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:26,349 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:26,349 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:26,349 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:26,349 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:26,352 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:26,352 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:26,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:26,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:26,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:26,353 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:26,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:26,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:26,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:26,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:26,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:26,354 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:26,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:26,355 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:26,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,358 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:26,358 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:26,358 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,358 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:26,358 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:26,358 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,358 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:26,358 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:26,358 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:26,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:26,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:26,358 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:26,358 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:26,358 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:26,358 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:26,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:26,359 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:26,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:26,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:26,459 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:26,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:26,461 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:26,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:26,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:26,463 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:26,463 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:26,463 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:26,463 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:26,463 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:26,463 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:26,464 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,464 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:26,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:26,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:26,465 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,465 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:26,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:26,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:26,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:26,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:26,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:26,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:26,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:26,466 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:26,466 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:26,466 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:26,466 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:26,466 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:26,466 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:26,466 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:26,467 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-21 01:56:26,467 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:26,467 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:26,467 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:26,467 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:26,467 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:26,467 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:26,490 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304219a27230ceebb4a3eaadd05cdc0098589 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304219a27230ceebb4a3eaadd05cdc0098589 2023-04-21 01:56:26,596 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:26,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:26,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:26,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042246610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682042186608, max timestamp=1682042186613Num uniques 859; 2023-04-21 01:56:26,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:56:26,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682042186608, max timestamp=1682042186613Num uniques 859; 2023-04-21 01:56:26,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:56:26,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682042186608, max timestamp=1682042186613Num uniques 859; 2023-04-21 01:56:26,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-21 01:56:26,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:26,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042246642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:26,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042246643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:26,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042246715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:26,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042246745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,749 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:26,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59372 deadline: 1682042246748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,895 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=502, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/251b64bcdfba45f7b87c73634e2f4229 2023-04-21 01:56:26,905 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/356449ebf565424f92e69191defd18e3 2023-04-21 01:56:26,914 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/547e25477c37476094b37a9c80795ae7 2023-04-21 01:56:26,918 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/251b64bcdfba45f7b87c73634e2f4229 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/251b64bcdfba45f7b87c73634e2f4229 2023-04-21 01:56:26,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:26,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042246918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,922 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/251b64bcdfba45f7b87c73634e2f4229, entries=150, sequenceid=502, filesize=25.4 K 2023-04-21 01:56:26,923 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/356449ebf565424f92e69191defd18e3 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/356449ebf565424f92e69191defd18e3 2023-04-21 01:56:26,926 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/356449ebf565424f92e69191defd18e3, entries=150, sequenceid=502, filesize=11.9 K 2023-04-21 01:56:26,927 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/547e25477c37476094b37a9c80795ae7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/547e25477c37476094b37a9c80795ae7 2023-04-21 01:56:26,929 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/547e25477c37476094b37a9c80795ae7, entries=150, sequenceid=502, filesize=11.9 K 2023-04-21 01:56:26,930 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 3db499d69ebc07e344a9c8f0976f4e89 in 464ms, sequenceid=502, compaction requested=true 2023-04-21 01:56:26,930 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:26,930 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:26,930 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:26,930 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:26,930 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,930 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:26,931 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:26,932 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:26,932 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:26,932 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:26,932 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:26,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:26,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fae1683[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:26,933 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:26,933 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:26,938 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:26,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:26,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:26,938 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:26,938 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:26,938 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:26,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,939 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:26,939 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,939 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:26,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:26,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:26,939 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:26,940 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:26,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:26,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:26,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:26,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:26,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:26,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:26,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:26,942 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:26,942 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:26,942 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:26,942 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:26,942 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:26,946 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:26,946 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:26,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:26,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:26,946 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:26,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:26,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:26,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:26,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:26,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:26,946 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:26,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:26,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:26,947 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,949 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,949 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:26,949 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:26,949 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,949 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:26,949 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:26,949 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:26,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:26,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:26,949 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:26,949 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:26,949 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:26,950 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:26,950 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:26,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:26,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:26,950 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:26,950 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:26,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:26,953 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-21 01:56:26,953 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:26,953 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:26,953 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:26,953 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:26,953 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:26,953 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:26,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:26,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042246965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:26,971 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:26,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59372 deadline: 1682042246968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,050 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:27,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:27,052 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:27,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:27,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:27,054 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:27,054 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:27,054 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:27,054 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:27,054 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,054 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,055 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,055 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:27,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:27,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:27,056 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,056 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:27,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:27,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:27,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:27,057 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:27,057 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:27,057 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:27,057 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:27,057 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:27,057 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:27,057 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:27,057 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:27,058 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:27,058 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:27,058 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:27,058 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,058 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:27,058 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:27,060 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,060 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:27,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:27,060 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,061 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:27,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:27,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:27,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:27,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:27,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6544fb1a[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:27,062 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:27,062 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,063 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,063 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:27,063 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:27,063 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:27,063 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,064 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:27,064 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:27,064 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:27,064 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,064 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:27,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:27,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:27,067 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:27,067 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:27,067 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:27,067 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:27,067 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:27,068 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,068 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:27,068 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:27,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:27,069 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,071 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:27,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,072 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:27,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042247070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:27,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59372 deadline: 1682042247072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,075 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,075 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,075 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:27,075 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:27,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:27,075 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:27,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:27,075 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:27,075 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:27,075 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:27,075 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,075 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:27,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:27,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:27,075 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:27,075 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,076 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,076 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,176 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:27,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:27,178 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:27,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:27,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:27,180 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:27,180 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:27,180 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:27,180 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:27,180 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,180 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,182 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,182 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:27,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:27,182 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,182 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:27,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:27,183 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:27,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:27,183 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:27,183 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:27,183 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:27,184 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:27,184 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:27,184 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:27,184 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:27,184 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:27,184 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:27,184 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:27,184 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:27,184 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,184 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:27,184 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:27,185 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,185 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:27,186 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,186 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:27,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:27,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:27,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20fa1c20[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:27,188 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:27,188 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,189 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:27,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:27,190 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:27,190 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,190 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:27,193 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:27,193 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:27,193 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:27,193 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:27,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:27,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:27,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:27,196 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:27,196 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:27,196 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:27,196 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:27,196 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:27,198 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:27,198 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,198 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:27,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:27,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:27,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:27,199 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,200 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:27,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,203 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,203 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,203 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,203 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:27,204 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:27,204 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:27,204 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,204 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,204 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,204 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,203 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:27,204 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:27,204 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:27,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:27,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:27,204 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:27,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:27,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:27,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:27,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042247222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:27,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042247274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:27,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59372 deadline: 1682042247275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,304 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:27,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:27,307 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:27,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:27,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:27,308 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:27,308 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:27,308 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:27,308 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:27,309 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,309 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,310 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,310 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:27,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:27,311 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,311 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:27,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:27,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:27,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:27,312 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:27,312 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:27,312 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:27,312 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:27,312 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:27,312 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:27,312 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:27,312 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:27,313 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:27,313 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:27,313 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:27,313 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,313 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:27,313 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:27,315 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:27,316 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:27,316 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,316 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:27,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:27,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74927250[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:27,317 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:27,317 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,318 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,318 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:27,318 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:27,318 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:27,319 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,319 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:27,320 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:27,320 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:27,320 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:27,320 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:27,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:27,322 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:27,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:27,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:27,322 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:27,322 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:27,324 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:27,324 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,324 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:27,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:27,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:27,324 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,326 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:27,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,338 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:27,338 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:27,338 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:27,338 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:27,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:27,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:27,338 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:27,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:27,339 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:27,338 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,339 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:27,339 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,339 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:27,339 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:27,339 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,339 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,339 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,365 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421dbd321774030424fa320f3bb4f1f7bda to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421dbd321774030424fa320f3bb4f1f7bda 2023-04-21 01:56:27,439 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:27,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:27,441 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:27,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:27,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:27,442 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:27,442 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:27,442 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:27,442 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:27,443 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,443 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,444 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,444 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:27,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:27,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:27,444 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,444 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:27,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:27,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:27,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:27,446 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:27,446 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:27,446 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:27,446 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:27,446 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:27,446 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:27,446 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:27,446 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:27,446 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:27,446 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:27,446 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:27,446 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:27,446 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,446 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:27,446 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:27,447 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,447 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:27,448 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:27,448 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:27,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:27,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bc7784f[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:27,449 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:27,449 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,452 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,452 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:27,452 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:27,452 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,452 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:27,452 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:27,453 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,453 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:27,453 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:27,453 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:27,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:27,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:27,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:27,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:27,456 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:27,456 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:27,456 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:27,456 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:27,456 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:27,457 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,457 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:27,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:27,457 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:27,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:27,458 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:27,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:27,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:27,459 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:27,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,461 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:27,461 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:27,461 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,461 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,461 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:27,461 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:27,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:27,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:27,461 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,461 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:27,462 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,462 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,462 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,462 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,462 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:27,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:27,462 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:27,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:27,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:27,562 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:27,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:27,564 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:27,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:27,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:27,565 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:27,566 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:27,566 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:27,566 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:27,566 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,566 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,568 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:27,568 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:27,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:27,568 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,568 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:27,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:27,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:27,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:27,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:27,569 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:27,570 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:27,570 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:27,570 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:27,570 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:27,570 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:27,570 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:27,570 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:27,570 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:27,572 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:27,572 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:27,572 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,572 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:27,572 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:27,573 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,573 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:27,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:27,574 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,574 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:27,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:27,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:27,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:27,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:27,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7999d806[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:27,576 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:27,576 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,577 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,577 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:27,577 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:27,577 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:27,577 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,577 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:27,578 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,578 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:27,578 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:27,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:27,578 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:27,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:27,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:27,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:27,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:27,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:27,581 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:27,581 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:27,581 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:27,581 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:27,581 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:27,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:27,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59372 deadline: 1682042247580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:27,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042247580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,584 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,584 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,584 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:27,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,585 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:27,586 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:27,588 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:27,588 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:27,588 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,588 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,588 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,588 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:27,588 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:27,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:27,588 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,588 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,588 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,588 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:27,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:27,588 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:27,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:27,588 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:27,589 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:27,689 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:27,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:27,691 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:27,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:27,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:27,692 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:27,692 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:27,692 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:27,692 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:27,693 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,693 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,694 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:27,694 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:27,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:27,694 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,694 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:27,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:27,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:27,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:27,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:27,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:27,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:27,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:27,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:27,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:27,696 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:27,696 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:27,696 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:27,696 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:27,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:27,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:27,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:27,696 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:27,698 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,698 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:27,698 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,698 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:27,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:27,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:27,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9f2d009[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:27,699 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:27,700 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,701 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,702 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:27,702 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:27,702 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:27,702 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,702 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:27,703 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,703 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:27,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:27,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:27,703 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:27,703 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:27,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:27,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:27,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:27,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:27,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:27,705 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:27,705 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:27,705 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:27,705 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:27,705 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:27,706 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:27,707 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,707 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:27,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:27,707 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:27,709 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,711 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,711 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,711 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,711 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:27,711 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:27,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:27,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:27,711 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,711 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,711 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:27,711 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:27,711 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:27,711 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,711 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:27,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:27,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:27,711 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:27,712 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:27,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:27,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042247726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,770 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=537, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/9f3ac2aea3fa4cfeb31a2b098994f740 2023-04-21 01:56:27,779 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/c2032ef0742d4fe086df3db9b9d1257e 2023-04-21 01:56:27,790 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/eeb560185b83484287edcafcd41038ca 2023-04-21 01:56:27,794 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/9f3ac2aea3fa4cfeb31a2b098994f740 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/9f3ac2aea3fa4cfeb31a2b098994f740 2023-04-21 01:56:27,798 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/9f3ac2aea3fa4cfeb31a2b098994f740, entries=200, sequenceid=537, filesize=32.3 K 2023-04-21 01:56:27,799 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/c2032ef0742d4fe086df3db9b9d1257e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/c2032ef0742d4fe086df3db9b9d1257e 2023-04-21 01:56:27,802 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/c2032ef0742d4fe086df3db9b9d1257e, entries=150, sequenceid=537, filesize=11.9 K 2023-04-21 01:56:27,803 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/eeb560185b83484287edcafcd41038ca as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/eeb560185b83484287edcafcd41038ca 2023-04-21 01:56:27,806 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/eeb560185b83484287edcafcd41038ca, entries=150, sequenceid=537, filesize=11.9 K 2023-04-21 01:56:27,806 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 3db499d69ebc07e344a9c8f0976f4e89 in 853ms, sequenceid=537, compaction requested=true 2023-04-21 01:56:27,807 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:27,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:27,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:27,807 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:56:27,807 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:56:27,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:27,809 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 112178 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:56:27,809 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/A is initiating minor compaction (all files) 2023-04-21 01:56:27,809 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/A in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:27,809 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/fd7040044eff43f3a7007256de9a9964, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/65bcdff729d74de882c7f6d2015f44bd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/251b64bcdfba45f7b87c73634e2f4229, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/9f3ac2aea3fa4cfeb31a2b098994f740] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=109.5 K 2023-04-21 01:56:27,809 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:56:27,809 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting fd7040044eff43f3a7007256de9a9964, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1682042184984 2023-04-21 01:56:27,809 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/B is initiating minor compaction (all files) 2023-04-21 01:56:27,809 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/B in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:27,809 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b75a9dece28c458b93e00081b0f7d080, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/07004024efd44692bcacc62889e20de3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/356449ebf565424f92e69191defd18e3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/c2032ef0742d4fe086df3db9b9d1257e] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=48.5 K 2023-04-21 01:56:27,810 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 65bcdff729d74de882c7f6d2015f44bd, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1682042185355 2023-04-21 01:56:27,810 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting b75a9dece28c458b93e00081b0f7d080, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1682042184984 2023-04-21 01:56:27,810 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 251b64bcdfba45f7b87c73634e2f4229, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1682042185477 2023-04-21 01:56:27,810 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 07004024efd44692bcacc62889e20de3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1682042185355 2023-04-21 01:56:27,810 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 9f3ac2aea3fa4cfeb31a2b098994f740, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1682042186622 2023-04-21 01:56:27,810 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 356449ebf565424f92e69191defd18e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1682042185477 2023-04-21 01:56:27,811 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting c2032ef0742d4fe086df3db9b9d1257e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1682042186622 2023-04-21 01:56:27,812 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:27,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:27,815 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:27,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:27,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:27,816 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:27,816 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:27,816 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:27,816 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:27,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,820 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#B#compaction#629 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:27,821 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:27,821 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:27,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:27,822 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:27,822 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:27,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:27,823 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:27,825 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#A#compaction#628 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:27,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:27,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:27,836 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:27,836 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:27,836 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:27,840 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:27,840 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:27,840 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:27,840 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:27,840 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-21 01:56:27,840 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:27,840 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:27,841 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:27,841 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:27,841 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:27,841 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:27,844 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/79c3b366832c4062a76fa97f0394396c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/79c3b366832c4062a76fa97f0394396c 2023-04-21 01:56:27,849 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/B of 3db499d69ebc07e344a9c8f0976f4e89 into 79c3b366832c4062a76fa97f0394396c(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:27,849 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:27,849 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/B, priority=12, startTime=1682042187807; duration=0sec 2023-04-21 01:56:27,849 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:27,849 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:56:27,850 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49664 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:56:27,850 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/C is initiating minor compaction (all files) 2023-04-21 01:56:27,850 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/C in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:27,850 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/9083e003a7fd4ba2ae67fc6969575cb7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/d96b4358870f4606bcd24e350bdc4e52, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/547e25477c37476094b37a9c80795ae7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/eeb560185b83484287edcafcd41038ca] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=48.5 K 2023-04-21 01:56:27,851 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 9083e003a7fd4ba2ae67fc6969575cb7, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1682042184984 2023-04-21 01:56:27,851 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting d96b4358870f4606bcd24e350bdc4e52, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1682042185355 2023-04-21 01:56:27,851 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 547e25477c37476094b37a9c80795ae7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1682042185477 2023-04-21 01:56:27,852 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting eeb560185b83484287edcafcd41038ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1682042186622 2023-04-21 01:56:27,878 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#C#compaction#631 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:28,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:28,089 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:28,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:28,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042248115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:28,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042188100, max timestamp=1682042188117Num uniques 859; 2023-04-21 01:56:28,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:28,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042188100, max timestamp=1682042188117Num uniques 859; 2023-04-21 01:56:28,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:28,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042188100, max timestamp=1682042188117Num uniques 859; 2023-04-21 01:56:28,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:28,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:28,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59372 deadline: 1682042248181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:28,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:28,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042248219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:28,274 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421b43024c842ed45ab91b8c9ded44edcca to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421b43024c842ed45ab91b8c9ded44edcca 2023-04-21 01:56:28,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:28,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59372 deadline: 1682042248284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:28,292 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/a25b5161b381478cab64040638fc9382 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/a25b5161b381478cab64040638fc9382 2023-04-21 01:56:28,292 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/7e63469621814a87ad3fe3cac6a26e79 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/7e63469621814a87ad3fe3cac6a26e79 2023-04-21 01:56:28,296 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/C of 3db499d69ebc07e344a9c8f0976f4e89 into a25b5161b381478cab64040638fc9382(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:28,296 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:28,296 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/A of 3db499d69ebc07e344a9c8f0976f4e89 into 7e63469621814a87ad3fe3cac6a26e79(size=26.5 K), total size for store is 26.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:28,296 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/C, priority=12, startTime=1682042187807; duration=0sec 2023-04-21 01:56:28,296 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:28,296 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/A, priority=12, startTime=1682042187807; duration=0sec 2023-04-21 01:56:28,296 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:28,296 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:28,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:28,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042248421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:28,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:28,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59372 deadline: 1682042248487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:28,679 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=548, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/dfe27bbf51ac4c7eb4bf0e853133b81e 2023-04-21 01:56:28,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:28,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042248725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:28,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:28,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042248736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:28,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:28,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59372 deadline: 1682042248791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,090 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/b85c0503e0524af6a22827f937241b04 2023-04-21 01:56:29,102 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/1b63b346cf124c4cb878a1c3f3358337 2023-04-21 01:56:29,105 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/dfe27bbf51ac4c7eb4bf0e853133b81e as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/dfe27bbf51ac4c7eb4bf0e853133b81e 2023-04-21 01:56:29,109 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/dfe27bbf51ac4c7eb4bf0e853133b81e, entries=150, sequenceid=548, filesize=25.4 K 2023-04-21 01:56:29,109 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/b85c0503e0524af6a22827f937241b04 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b85c0503e0524af6a22827f937241b04 2023-04-21 01:56:29,113 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b85c0503e0524af6a22827f937241b04, entries=150, sequenceid=548, filesize=11.9 K 2023-04-21 01:56:29,113 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/1b63b346cf124c4cb878a1c3f3358337 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/1b63b346cf124c4cb878a1c3f3358337 2023-04-21 01:56:29,116 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/1b63b346cf124c4cb878a1c3f3358337, entries=150, sequenceid=548, filesize=11.9 K 2023-04-21 01:56:29,117 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 3db499d69ebc07e344a9c8f0976f4e89 in 1277ms, sequenceid=548, compaction requested=false 2023-04-21 01:56:29,117 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:29,117 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:29,117 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:29,117 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:29,117 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,117 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:29,117 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:29,119 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:29,119 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:29,119 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,119 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:29,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:29,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:29,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:29,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e952395[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:29,122 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:29,122 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,122 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,123 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:29,123 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:29,123 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:29,123 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,123 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:29,125 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,125 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:29,125 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:29,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:29,125 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:29,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:29,128 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:29,128 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:29,128 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:29,128 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:29,128 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:29,130 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,130 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,130 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:29,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:29,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:29,131 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:29,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:29,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:29,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,132 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,134 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:29,134 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,134 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,134 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,134 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,134 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,134 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,135 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,134 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:29,135 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:29,135 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:29,135 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:29,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:29,135 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:29,135 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:29,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:29,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:29,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:29,232 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-21 01:56:29,233 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:29,233 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:29,233 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:29,233 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:29,233 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:29,233 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:29,235 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:29,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:29,237 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:29,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:29,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:29,239 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:29,239 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:29,239 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:29,239 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:29,240 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,240 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,241 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:29,241 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:29,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:29,242 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,242 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:29,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:29,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:29,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:29,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:29,244 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:29,244 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:29,244 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:29,245 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:29,245 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:29,245 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:29,245 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:29,245 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:29,245 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:29,245 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:29,245 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:29,245 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,245 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:29,245 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:29,248 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,248 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:29,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:29,249 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,249 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:29,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:29,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e102b08[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:29,252 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:29,252 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,253 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,253 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:29,253 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:29,253 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,253 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:29,253 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:29,254 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:29,254 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,255 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:29,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:29,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:29,255 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:29,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:29,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:29,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:29,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:29,258 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:29,258 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:29,258 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:29,258 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:29,258 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:29,259 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,259 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:29,260 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:29,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:29,260 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:29,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042249256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,262 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,265 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,265 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,265 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,265 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:29,265 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,265 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:29,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:29,265 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:29,265 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:29,265 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:29,265 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,265 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,265 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:29,265 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:29,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:29,265 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:29,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:29,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:29,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:29,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59372 deadline: 1682042249296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:29,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042249362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,366 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:29,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:29,368 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:29,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:29,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:29,369 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:29,369 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:29,370 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:29,370 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:29,370 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,370 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,371 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:29,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:29,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:29,371 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,372 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,372 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:29,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:29,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:29,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:29,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:29,373 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:29,373 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:29,373 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:29,373 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:29,373 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:29,373 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:29,373 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:29,373 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:29,374 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:29,374 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:29,374 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:29,374 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,374 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:29,374 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:29,376 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:29,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:29,376 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:29,377 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,377 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:29,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:29,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59fe8724[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:29,378 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:29,378 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,379 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,380 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,380 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:29,380 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:29,380 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:29,380 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:29,382 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,382 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:29,382 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:29,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:29,382 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:29,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:29,385 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:29,385 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:29,385 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:29,385 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:29,385 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:29,386 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,387 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:29,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,387 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:29,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:29,387 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:29,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,388 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,391 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:29,391 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:29,391 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:29,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:29,391 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:29,391 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:29,391 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:29,391 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,392 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,392 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,392 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,392 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,392 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,392 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:29,392 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:29,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:29,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:29,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:29,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59426 deadline: 1682042249399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,402 DEBUG [Thread-4045] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., hostname=jenkins-hbase4.apache.org,40131,1682042019399, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-21 01:56:29,492 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:29,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:29,494 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:29,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:29,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:29,495 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:29,495 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:29,495 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:29,495 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:29,496 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,496 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,497 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:29,497 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:29,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:29,497 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,497 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:29,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:29,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:29,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:29,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:29,498 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:29,498 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:29,499 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:29,499 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:29,499 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:29,499 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:29,499 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:29,499 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:29,499 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:29,499 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:29,499 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:29,499 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,499 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:29,499 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:29,504 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,504 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:29,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:29,504 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:29,504 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:29,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:29,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:29,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4613668a[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:29,505 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:29,505 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,506 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,507 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:29,507 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:29,507 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:29,507 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,507 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:29,508 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:29,508 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:29,508 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:29,509 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:29,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:29,511 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:29,511 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:29,511 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:29,511 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:29,511 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:29,513 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,513 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,513 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:29,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,514 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,515 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,518 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,518 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,518 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,518 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,518 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:29,518 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:29,518 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:29,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:29,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:29,518 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:29,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:29,518 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:29,518 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:29,518 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:29,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:29,518 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:29,518 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,518 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:29,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042249565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,618 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:29,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:29,620 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:29,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:29,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:29,622 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:29,622 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:29,622 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:29,622 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:29,622 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,622 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,623 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:29,623 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:29,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:29,624 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,624 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:29,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:29,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:29,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:29,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:29,625 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:29,625 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:29,625 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:29,625 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:29,625 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:29,625 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:29,625 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:29,625 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:29,625 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:29,625 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:29,626 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:29,626 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,626 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:29,626 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:29,627 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,627 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:29,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:29,627 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,627 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:29,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:29,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15d7c56[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:29,629 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:29,629 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,630 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,630 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:29,630 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:29,630 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:29,631 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,631 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:29,633 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:29,633 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:29,633 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,633 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:29,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:29,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:29,635 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:29,635 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:29,635 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:29,635 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:29,635 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:29,637 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,637 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,637 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:29,637 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:29,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:29,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:29,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:29,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,639 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,641 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,641 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,641 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,641 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,641 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,641 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:29,641 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:29,641 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:29,641 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:29,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:29,641 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:29,641 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:29,641 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,641 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:29,641 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:29,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:29,653 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421f20f74a3717d4d47a6f33009cb87ee3d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421f20f74a3717d4d47a6f33009cb87ee3d 2023-04-21 01:56:29,657 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=588, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/a579fd93cd7c42489b5aaf437b5a1791 2023-04-21 01:56:29,666 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=588 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/07e855a60d3c4800b3f4c12377b1c2d7 2023-04-21 01:56:29,742 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:29,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:29,744 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:29,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:29,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:29,745 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:29,745 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:29,745 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:29,745 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:29,746 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,746 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,747 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:29,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,748 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,749 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,749 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:29,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:29,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:29,749 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:29,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:29,749 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:29,749 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:29,749 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:29,750 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:29,750 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:29,750 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:29,750 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:29,750 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:29,750 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:29,750 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:29,750 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:29,750 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,750 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:29,750 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:29,753 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,753 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:29,753 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,753 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:29,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:29,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:29,755 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:29,755 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e7a5f1e[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:29,756 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,756 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:29,756 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:29,756 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:29,756 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,756 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:29,757 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:29,757 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:29,757 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:29,757 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:29,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:29,760 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:29,760 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:29,760 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:29,760 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:29,760 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:29,761 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,761 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,761 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:29,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:29,762 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,763 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,766 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,766 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,766 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,766 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,766 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,766 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:29,767 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,767 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,767 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:29,767 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:29,766 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:29,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:29,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:29,767 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:29,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:29,767 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:29,768 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:29,868 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:29,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:29,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:29,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042249867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,870 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:29,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:29,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:29,872 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:29,872 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:29,872 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:29,872 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:29,872 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,872 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,873 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,873 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:29,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:29,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:29,874 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,874 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:29,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:29,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:29,875 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:29,875 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:29,875 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:29,875 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:29,875 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:29,875 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:29,875 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:29,875 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:29,875 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:29,875 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:29,875 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:29,875 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,875 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:29,875 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:29,877 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,877 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:29,877 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,877 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:29,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:29,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:29,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43103252[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:29,878 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:29,879 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,880 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:29,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:29,880 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:29,880 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,880 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:29,885 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:29,885 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:29,885 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:29,885 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:29,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:29,887 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:29,887 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:29,887 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:29,887 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:29,888 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:29,889 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,889 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,889 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:29,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,890 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:29,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:29,891 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:29,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:29,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,892 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,893 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,893 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,893 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:29,893 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:29,893 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:29,893 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:29,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:29,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:29,893 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:29,893 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:29,893 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,893 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,893 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:29,893 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,893 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:29,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:29,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:29,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:29,993 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:29,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:29,995 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:29,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:29,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:29,997 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:29,997 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:29,997 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:29,997 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:29,997 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:29,998 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,999 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:29,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:29,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:29,999 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:29,999 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:29,999 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:30,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:30,000 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:30,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:30,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:30,000 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:30,000 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:30,001 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:30,001 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:30,001 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:30,001 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:30,001 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:30,001 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:30,001 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:30,001 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:30,001 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:30,001 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,001 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:30,001 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:30,003 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:30,003 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:30,004 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,004 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:30,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:30,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e6a3b8c[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:30,005 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:30,005 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,006 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,006 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:30,007 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:30,007 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:30,007 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,007 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:30,008 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:30,008 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:30,008 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,008 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:30,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:30,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:30,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:30,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:30,011 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:30,011 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:30,011 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:30,011 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:30,011 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:30,013 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,013 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:30,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,013 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:30,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:30,014 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,015 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,018 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,018 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:30,018 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:30,018 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:30,018 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:30,018 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:30,019 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:30,019 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:30,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:30,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:30,019 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:30,018 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,019 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:30,019 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,019 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,019 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:30,076 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=588 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/28213a91e4544bc59712ad1e340ac180 2023-04-21 01:56:30,080 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/a579fd93cd7c42489b5aaf437b5a1791 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/a579fd93cd7c42489b5aaf437b5a1791 2023-04-21 01:56:30,085 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/a579fd93cd7c42489b5aaf437b5a1791, entries=200, sequenceid=588, filesize=32.3 K 2023-04-21 01:56:30,085 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/07e855a60d3c4800b3f4c12377b1c2d7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/07e855a60d3c4800b3f4c12377b1c2d7 2023-04-21 01:56:30,089 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/07e855a60d3c4800b3f4c12377b1c2d7, entries=150, sequenceid=588, filesize=11.9 K 2023-04-21 01:56:30,089 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/28213a91e4544bc59712ad1e340ac180 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/28213a91e4544bc59712ad1e340ac180 2023-04-21 01:56:30,092 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/28213a91e4544bc59712ad1e340ac180, entries=150, sequenceid=588, filesize=11.9 K 2023-04-21 01:56:30,093 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 3db499d69ebc07e344a9c8f0976f4e89 in 861ms, sequenceid=588, compaction requested=true 2023-04-21 01:56:30,093 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:30,093 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:30,093 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:56:30,093 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:56:30,093 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:30,093 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:30,094 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 86268 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:56:30,094 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:56:30,094 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/B is initiating minor compaction (all files) 2023-04-21 01:56:30,094 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/B in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:30,094 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/79c3b366832c4062a76fa97f0394396c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b85c0503e0524af6a22827f937241b04, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/07e855a60d3c4800b3f4c12377b1c2d7] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=36.8 K 2023-04-21 01:56:30,094 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/A is initiating minor compaction (all files) 2023-04-21 01:56:30,095 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/A in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:30,095 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/7e63469621814a87ad3fe3cac6a26e79, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/dfe27bbf51ac4c7eb4bf0e853133b81e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/a579fd93cd7c42489b5aaf437b5a1791] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=84.2 K 2023-04-21 01:56:30,095 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 79c3b366832c4062a76fa97f0394396c, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1682042186622 2023-04-21 01:56:30,095 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 7e63469621814a87ad3fe3cac6a26e79, keycount=150, bloomtype=ROW, size=26.5 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1682042186622 2023-04-21 01:56:30,095 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting b85c0503e0524af6a22827f937241b04, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1682042186960 2023-04-21 01:56:30,096 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting dfe27bbf51ac4c7eb4bf0e853133b81e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1682042186960 2023-04-21 01:56:30,096 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 07e855a60d3c4800b3f4c12377b1c2d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1682042188143 2023-04-21 01:56:30,096 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting a579fd93cd7c42489b5aaf437b5a1791, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1682042188143 2023-04-21 01:56:30,103 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#B#compaction#638 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:30,104 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#A#compaction#637 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:30,119 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:30,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:30,123 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:30,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:30,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:30,125 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:30,125 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:30,125 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:30,125 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:30,126 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,126 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:30,126 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/23eb6936fe454be486563f585572f571 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/23eb6936fe454be486563f585572f571 2023-04-21 01:56:30,127 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:30,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:30,127 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:30,127 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,128 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:30,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:30,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:30,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:30,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:30,130 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:30,130 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:30,130 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:30,131 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:30,131 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:30,131 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:30,131 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:30,131 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-21 01:56:30,132 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/B of 3db499d69ebc07e344a9c8f0976f4e89 into 23eb6936fe454be486563f585572f571(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:30,132 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:30,132 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/B, priority=13, startTime=1682042190093; duration=0sec 2023-04-21 01:56:30,132 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:30,132 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:56:30,133 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:30,133 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:30,133 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:30,133 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:30,133 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:30,133 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:30,133 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:56:30,133 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/C is initiating minor compaction (all files) 2023-04-21 01:56:30,133 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/C in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:30,134 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/a25b5161b381478cab64040638fc9382, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/1b63b346cf124c4cb878a1c3f3358337, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/28213a91e4544bc59712ad1e340ac180] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=36.7 K 2023-04-21 01:56:30,134 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting a25b5161b381478cab64040638fc9382, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1682042186622 2023-04-21 01:56:30,134 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 1b63b346cf124c4cb878a1c3f3358337, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1682042186960 2023-04-21 01:56:30,135 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 28213a91e4544bc59712ad1e340ac180, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1682042188143 2023-04-21 01:56:30,145 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#C#compaction#640 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:30,152 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421f74972bd03124d76930c013c206196ba to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421f74972bd03124d76930c013c206196ba 2023-04-21 01:56:30,169 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=599, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/6113929947344f1d8fb2732c6f9cd59c 2023-04-21 01:56:30,178 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/bea9270545184044985c09a6651b500c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/bea9270545184044985c09a6651b500c 2023-04-21 01:56:30,182 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/db94cd1c7778465d97487d9c4fba9ae6 2023-04-21 01:56:30,183 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/C of 3db499d69ebc07e344a9c8f0976f4e89 into bea9270545184044985c09a6651b500c(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:30,183 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:30,183 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/C, priority=13, startTime=1682042190093; duration=0sec 2023-04-21 01:56:30,183 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:30,192 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/06f0d56e7d7f4b51b233e38eb2d01c28 2023-04-21 01:56:30,195 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/6113929947344f1d8fb2732c6f9cd59c as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/6113929947344f1d8fb2732c6f9cd59c 2023-04-21 01:56:30,199 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/6113929947344f1d8fb2732c6f9cd59c, entries=150, sequenceid=599, filesize=25.4 K 2023-04-21 01:56:30,200 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/db94cd1c7778465d97487d9c4fba9ae6 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/db94cd1c7778465d97487d9c4fba9ae6 2023-04-21 01:56:30,204 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/db94cd1c7778465d97487d9c4fba9ae6, entries=150, sequenceid=599, filesize=11.9 K 2023-04-21 01:56:30,205 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/06f0d56e7d7f4b51b233e38eb2d01c28 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/06f0d56e7d7f4b51b233e38eb2d01c28 2023-04-21 01:56:30,208 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/06f0d56e7d7f4b51b233e38eb2d01c28, entries=150, sequenceid=599, filesize=11.9 K 2023-04-21 01:56:30,209 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 3db499d69ebc07e344a9c8f0976f4e89 in 78ms, sequenceid=599, compaction requested=false 2023-04-21 01:56:30,209 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:30,209 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:30,209 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:30,209 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:30,209 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,209 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:30,209 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:30,211 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:30,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:30,211 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:30,211 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,211 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:30,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:30,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:30,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:30,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@743f5bdb[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:30,213 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:30,213 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,214 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,214 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:30,214 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:30,214 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:30,214 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,214 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:30,217 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,217 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:30,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:30,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:30,217 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:30,217 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:30,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:30,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:30,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:30,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:30,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:30,219 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:30,219 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:30,219 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:30,219 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:30,219 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:30,221 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,221 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:30,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:30,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,221 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:30,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:30,221 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:30,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:30,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:30,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,223 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:30,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,225 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:30,225 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:30,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:30,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:30,225 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,225 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:30,225 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:30,225 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:30,225 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:30,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:30,226 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:30,225 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:30,226 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:30,226 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,226 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,226 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,226 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:30,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:30,326 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:30,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:30,328 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:30,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:30,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:30,330 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:30,330 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:30,330 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:30,330 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:30,330 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,330 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:30,331 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,331 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:30,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:30,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:30,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:30,332 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,332 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:30,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:30,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:30,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:30,333 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:30,333 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:30,333 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:30,333 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:30,333 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:30,333 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:30,333 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:30,334 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:30,334 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:30,334 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-21 01:56:30,334 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:30,334 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:30,334 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:30,334 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:30,334 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:30,334 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:30,345 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421006b1b78357b443a86d1c270293f09d2 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421006b1b78357b443a86d1c270293f09d2 2023-04-21 01:56:30,350 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=605, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/c0d59d8d2fad4365b066785fbbbd2799 2023-04-21 01:56:30,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:30,360 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:30,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042190378, max timestamp=1682042190382Num uniques 859; 2023-04-21 01:56:30,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:30,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042190378, max timestamp=1682042190382Num uniques 859; 2023-04-21 01:56:30,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:30,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682042190378, max timestamp=1682042190382Num uniques 859; 2023-04-21 01:56:30,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-21 01:56:30,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682042190407, max timestamp=1682042190412Num uniques 1018; 2023-04-21 01:56:30,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-21 01:56:30,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682042190407, max timestamp=1682042190412Num uniques 1018; 2023-04-21 01:56:30,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-21 01:56:30,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682042190407, max timestamp=1682042190412Num uniques 1009; 2023-04-21 01:56:30,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-21 01:56:30,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682042190436, max timestamp=1682042190444Num uniques 1027; 2023-04-21 01:56:30,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-21 01:56:30,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682042190439, max timestamp=1682042190444Num uniques 1027; 2023-04-21 01:56:30,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-21 01:56:30,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682042190439, max timestamp=1682042190444Num uniques 1027; 2023-04-21 01:56:30,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-21 01:56:30,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682042190468, max timestamp=1682042190478Num uniques 1086; 2023-04-21 01:56:30,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-21 01:56:30,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682042190468, max timestamp=1682042190478Num uniques 1009; 2023-04-21 01:56:30,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682042190468, max timestamp=1682042190478Num uniques 1036; 2023-04-21 01:56:30,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-21 01:56:30,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-21 01:56:30,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682042190491, max timestamp=1682042190509Num uniques 1045; 2023-04-21 01:56:30,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-21 01:56:30,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682042190491, max timestamp=1682042190509Num uniques 1045; 2023-04-21 01:56:30,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-21 01:56:30,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682042190491, max timestamp=1682042190509Num uniques 1045; 2023-04-21 01:56:30,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-21 01:56:30,533 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/3363451d07214bb599a2012e5b54ffe7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/3363451d07214bb599a2012e5b54ffe7 2023-04-21 01:56:30,537 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/A of 3db499d69ebc07e344a9c8f0976f4e89 into 3363451d07214bb599a2012e5b54ffe7(size=26.6 K), total size for store is 52.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:30,537 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:30,537 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/A, priority=13, startTime=1682042190093; duration=0sec 2023-04-21 01:56:30,537 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:30,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9336, totalHeapSize=21456, min timestamp=1682042190517, max timestamp=1682042190540Num uniques 1054; 2023-04-21 01:56:30,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48266, new segment data size=9336, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-21 01:56:30,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9334, totalHeapSize=21456, min timestamp=1682042190517, max timestamp=1682042190540Num uniques 1054; 2023-04-21 01:56:30,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48264, new segment data size=9334, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-21 01:56:30,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9334, totalHeapSize=21456, min timestamp=1682042190517, max timestamp=1682042190540Num uniques 1054; 2023-04-21 01:56:30,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48264, new segment data size=9334, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-21 01:56:30,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682042190556, max timestamp=1682042190566Num uniques 1063; 2023-04-21 01:56:30,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682042190556, max timestamp=1682042190566Num uniques 1063; 2023-04-21 01:56:30,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682042190556, max timestamp=1682042190566Num uniques 1063; 2023-04-21 01:56:30,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-21 01:56:30,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-21 01:56:30,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-21 01:56:30,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682042190578, max timestamp=1682042190591Num uniques 1022; 2023-04-21 01:56:30,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-21 01:56:30,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682042190578, max timestamp=1682042190591Num uniques 1022; 2023-04-21 01:56:30,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682042190578, max timestamp=1682042190591Num uniques 1022; 2023-04-21 01:56:30,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-21 01:56:30,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-21 01:56:30,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682042190612, max timestamp=1682042190617Num uniques 1081; 2023-04-21 01:56:30,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-21 01:56:30,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682042190612, max timestamp=1682042190617Num uniques 1081; 2023-04-21 01:56:30,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682042190612, max timestamp=1682042190617Num uniques 1081; 2023-04-21 01:56:30,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-21 01:56:30,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-21 01:56:30,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682042190635, max timestamp=1682042190645Num uniques 1040; 2023-04-21 01:56:30,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682042190635, max timestamp=1682042190645Num uniques 1040; 2023-04-21 01:56:30,648 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682042190635, max timestamp=1682042190645Num uniques 1040; 2023-04-21 01:56:30,648 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-21 01:56:30,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-21 01:56:30,648 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-21 01:56:30,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682042190663, max timestamp=1682042190671Num uniques 1049; 2023-04-21 01:56:30,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-21 01:56:30,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682042190663, max timestamp=1682042190671Num uniques 1049; 2023-04-21 01:56:30,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-21 01:56:30,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682042190663, max timestamp=1682042190671Num uniques 1049; 2023-04-21 01:56:30,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-21 01:56:30,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682042190701, max timestamp=1682042190706Num uniques 1108; 2023-04-21 01:56:30,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682042190689, max timestamp=1682042190706Num uniques 1108; 2023-04-21 01:56:30,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-21 01:56:30,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-21 01:56:30,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682042190701, max timestamp=1682042190706Num uniques 1108; 2023-04-21 01:56:30,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-21 01:56:30,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1682042190727, max timestamp=1682042190733Num uniques 1067; 2023-04-21 01:56:30,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-21 01:56:30,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1682042190727, max timestamp=1682042190733Num uniques 1017; 2023-04-21 01:56:30,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1682042190727, max timestamp=1682042190733Num uniques 1017; 2023-04-21 01:56:30,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-21 01:56:30,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-21 01:56:30,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682042190744, max timestamp=1682042190757Num uniques 1026; 2023-04-21 01:56:30,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-21 01:56:30,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682042190744, max timestamp=1682042190757Num uniques 1026; 2023-04-21 01:56:30,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682042190744, max timestamp=1682042190757Num uniques 1026; 2023-04-21 01:56:30,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-21 01:56:30,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-21 01:56:30,770 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/f8eefe0508194bd38c9f185e462b068f 2023-04-21 01:56:30,785 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/8c04156a1a764c3b9f9d3db60aad854b 2023-04-21 01:56:30,786 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1682042190780, max timestamp=1682042190784Num uniques 1035; 2023-04-21 01:56:30,786 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-21 01:56:30,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8461, totalHeapSize=19480, min timestamp=1682042190780, max timestamp=1682042190785Num uniques 1035; 2023-04-21 01:56:30,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47391, new segment data size=8461, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-21 01:56:30,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1682042190780, max timestamp=1682042190784Num uniques 1035; 2023-04-21 01:56:30,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-21 01:56:30,794 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/c0d59d8d2fad4365b066785fbbbd2799 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/c0d59d8d2fad4365b066785fbbbd2799 2023-04-21 01:56:30,797 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/c0d59d8d2fad4365b066785fbbbd2799, entries=100, sequenceid=605, filesize=18.6 K 2023-04-21 01:56:30,798 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/f8eefe0508194bd38c9f185e462b068f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f8eefe0508194bd38c9f185e462b068f 2023-04-21 01:56:30,803 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f8eefe0508194bd38c9f185e462b068f, entries=50, sequenceid=605, filesize=7.1 K 2023-04-21 01:56:30,804 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/8c04156a1a764c3b9f9d3db60aad854b as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/8c04156a1a764c3b9f9d3db60aad854b 2023-04-21 01:56:30,807 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/8c04156a1a764c3b9f9d3db60aad854b, entries=50, sequenceid=605, filesize=7.1 K 2023-04-21 01:56:30,808 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=134.18 KB/137400 for 3db499d69ebc07e344a9c8f0976f4e89 in 474ms, sequenceid=605, compaction requested=true 2023-04-21 01:56:30,808 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:30,808 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:30,808 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:30,809 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:30,809 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,809 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:30,809 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:30,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:30,809 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=140.89 KB heapSize=403.17 KB 2023-04-21 01:56:30,809 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:30,809 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-21 01:56:30,809 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:30,809 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-21 01:56:30,809 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:30,809 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-21 01:56:30,813 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:30,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:30,813 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,814 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,814 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:30,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:30,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:30,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:30,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:30,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@480178ed[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:30,816 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:30,816 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,817 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,817 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:30,817 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:30,817 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:30,817 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:30,819 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:30,819 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:30,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:30,819 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:30,819 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:30,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:30,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:30,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:30,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042250816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:30,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:30,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042250819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:30,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59372 deadline: 1682042250820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:30,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:30,824 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:30,824 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:30,824 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:30,824 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:30,825 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:30,826 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042134eb0ea03d1c44fe9665eb052c3f0f2d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042134eb0ea03d1c44fe9665eb052c3f0f2d 2023-04-21 01:56:30,830 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,830 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:30,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,830 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:30,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:30,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:30,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:30,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,831 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:30,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:30,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,833 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:30,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:30,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:30,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,835 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,838 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:30,838 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:30,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:30,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:30,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:30,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:30,838 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:30,838 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:30,838 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:30,839 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:30,839 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:30,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:30,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042250920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:30,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042250924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:30,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59372 deadline: 1682042250924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,939 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:30,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:30,941 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:30,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:30,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:30,942 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:30,942 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:30,942 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:30,942 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:30,942 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,942 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:30,943 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:30,944 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:30,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:30,944 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,944 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:30,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:30,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:30,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:30,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:30,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:30,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:30,945 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:30,945 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:30,945 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:30,945 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:30,945 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:30,945 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:30,945 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:30,945 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:30,945 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:30,945 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:30,945 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:30,945 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,946 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:30,946 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:30,947 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,947 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:30,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:30,947 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,947 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:30,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:30,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:30,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:30,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:30,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ac1411d[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:30,948 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:30,948 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,955 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,956 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:30,956 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:30,956 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:30,956 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,956 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:30,957 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,957 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:30,957 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:30,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:30,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:30,957 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:30,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:30,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:30,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:30,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:30,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:30,960 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:30,960 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:30,960 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:30,960 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:30,960 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:30,961 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,961 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:30,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,961 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:30,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:30,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:30,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:30,962 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:30,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:30,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,963 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:30,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:30,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:30,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:30,967 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:30,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:30,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:30,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:30,967 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:30,967 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:30,967 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:30,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:30,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:30,968 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:30,968 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:30,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:31,068 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:31,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:31,070 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:31,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:31,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:31,071 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:31,071 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:31,071 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:31,071 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:31,071 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,071 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,074 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:31,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:31,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:31,074 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,074 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,075 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:31,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:31,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:31,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:31,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:31,075 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:31,075 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:31,075 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:31,075 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:31,075 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:31,075 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:31,075 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:31,076 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:31,076 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:31,076 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:31,076 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:31,076 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,076 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:31,076 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:31,077 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,077 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:31,077 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,077 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:31,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:31,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:31,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:31,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45b8bd92[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:31,079 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:31,079 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,080 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,080 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:31,080 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:31,080 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:31,080 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,080 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:31,081 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,082 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:31,082 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:31,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:31,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:31,082 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:31,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:31,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:31,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:31,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:31,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:31,085 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:31,085 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:31,085 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:31,085 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:31,085 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:31,104 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,104 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:31,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,105 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,105 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:31,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,106 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,108 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:31,109 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:31,109 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,109 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,109 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:31,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:31,109 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:31,109 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:31,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:31,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:31,109 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,109 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,109 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,109 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,109 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:31,109 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:31,109 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:31,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:31,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:31,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042251123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:31,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:31,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042251128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59372 deadline: 1682042251128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,209 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:31,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:31,211 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:31,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:31,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:31,213 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:31,213 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:31,213 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:31,213 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:31,213 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,213 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,214 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:31,214 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:31,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:31,215 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,215 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:31,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:31,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:31,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:31,215 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:31,215 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:31,216 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:31,216 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:31,216 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:31,216 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:31,216 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:31,216 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:31,216 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:31,216 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:31,216 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:31,216 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,216 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:31,216 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:31,217 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,217 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:31,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:31,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:31,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:31,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:31,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:31,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bd7d77d[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:31,220 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:31,220 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,222 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,222 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:31,222 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:31,222 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:31,222 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,222 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:31,224 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:31,224 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:31,224 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:31,224 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:31,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:31,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:31,227 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:31,227 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:31,227 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:31,227 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:31,227 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:31,228 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:31,228 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,228 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:31,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:31,229 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,230 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,236 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,236 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,236 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,236 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,236 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,236 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,236 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,236 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:31,236 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:31,236 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:31,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:31,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:31,236 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:31,236 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:31,236 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:31,236 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:31,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:31,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:31,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:31,243 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=884, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/9488c294b9b04f6c87316deefe2572fe 2023-04-21 01:56:31,251 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=884 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/8b859d11d5144b8a9f743676a14ecee2 2023-04-21 01:56:31,336 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:31,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:31,339 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:31,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:31,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:31,340 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:31,340 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:31,340 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:31,340 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:31,341 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,341 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,342 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:31,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:31,342 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,342 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,342 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:31,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:31,343 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:31,343 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:31,343 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:31,343 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:31,343 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:31,344 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:31,344 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:31,344 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:31,344 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:31,344 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:31,344 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:31,344 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:31,344 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:31,344 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,344 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:31,344 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:31,346 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,346 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:31,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:31,346 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,346 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:31,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:31,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:31,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:31,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:31,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8d218fa[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:31,347 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:31,347 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,348 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,349 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:31,349 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,349 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:31,349 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:31,349 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:31,351 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,351 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:31,351 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:31,351 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:31,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:31,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:31,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:31,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:31,354 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:31,354 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:31,354 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:31,354 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:31,354 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:31,357 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,357 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,357 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:31,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:31,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:31,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:31,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:31,358 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:31,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:31,359 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:31,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,361 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:31,361 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:31,361 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,362 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,362 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:31,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:31,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:31,362 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:31,362 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:31,362 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:31,362 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,362 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,362 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,362 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:31,362 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:31,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:31,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:31,362 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:31,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042251427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:31,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59372 deadline: 1682042251435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:31,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042251435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,462 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:31,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:31,465 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:31,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:31,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:31,466 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:31,466 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:31,466 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:31,466 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:31,467 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,467 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,468 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:31,468 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:31,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:31,469 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,469 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:31,470 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:31,470 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:31,470 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:31,470 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:31,470 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:31,470 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:31,471 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:31,471 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:31,471 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:31,471 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:31,471 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:31,471 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:31,471 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:31,471 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,472 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:31,472 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:31,474 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,474 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:31,475 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,475 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:31,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:31,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@89b7954[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:31,476 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:31,476 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,477 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,478 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:31,478 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:31,478 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:31,478 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,478 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:31,479 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:31,479 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:31,479 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:31,479 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:31,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:31,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:31,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:31,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:31,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:31,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:31,482 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:31,482 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:31,482 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:31,482 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:31,482 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:31,484 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,484 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,484 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:31,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:31,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:31,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:31,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:31,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:31,485 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,486 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:31,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,492 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,492 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:31,492 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:31,492 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:31,493 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:31,492 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:31,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-21 01:56:31,492 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,493 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:31,493 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,493 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:31,493 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,493 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,493 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,493 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:31,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:31,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:31,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:31,593 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:31,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:31,595 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:31,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:31,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:31,596 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:31,596 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:31,596 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:31,596 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:31,597 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,597 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,598 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,598 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:31,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:31,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:31,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,598 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,599 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:31,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:31,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:31,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:31,599 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:31,600 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:31,600 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:31,600 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:31,600 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:31,600 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:31,600 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:31,600 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:31,600 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:31,600 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:31,600 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:31,600 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,600 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:31,600 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:31,601 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,601 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:31,602 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,602 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:31,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:31,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@84f11f8[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:31,603 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:31,603 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,604 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,604 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:31,604 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:31,604 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:31,604 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,604 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:31,606 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:31,606 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,606 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:31,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:31,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:31,607 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:31,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:31,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:31,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:31,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:31,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:31,609 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:31,609 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:31,609 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:31,609 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:31,609 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:31,611 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,611 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,611 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:31,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:31,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:31,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:31,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:31,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:31,611 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:31,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,613 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:31,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:31,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,615 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:31,615 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:31,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:31,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:31,615 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,615 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,615 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:31,615 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:31,615 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,615 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:31,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:31,615 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:31,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:31,615 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:31,615 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,616 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,616 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:31,616 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,661 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=884 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/91b48c23cea641eeae16fc97968bbdf5 2023-04-21 01:56:31,666 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/9488c294b9b04f6c87316deefe2572fe as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/9488c294b9b04f6c87316deefe2572fe 2023-04-21 01:56:31,669 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/9488c294b9b04f6c87316deefe2572fe, entries=200, sequenceid=884, filesize=32.3 K 2023-04-21 01:56:31,670 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/8b859d11d5144b8a9f743676a14ecee2 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/8b859d11d5144b8a9f743676a14ecee2 2023-04-21 01:56:31,674 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/8b859d11d5144b8a9f743676a14ecee2, entries=150, sequenceid=884, filesize=11.9 K 2023-04-21 01:56:31,674 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/91b48c23cea641eeae16fc97968bbdf5 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/91b48c23cea641eeae16fc97968bbdf5 2023-04-21 01:56:31,678 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/91b48c23cea641eeae16fc97968bbdf5, entries=150, sequenceid=884, filesize=11.9 K 2023-04-21 01:56:31,679 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~403.13 KB/412800, currentSize=40.25 KB/41220 for 3db499d69ebc07e344a9c8f0976f4e89 in 870ms, sequenceid=884, compaction requested=true 2023-04-21 01:56:31,679 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:31,679 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:31,679 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:56:31,679 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:56:31,679 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:31,679 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:31,680 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 105366 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:56:31,680 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45050 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:56:31,681 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/A is initiating minor compaction (all files) 2023-04-21 01:56:31,681 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/B is initiating minor compaction (all files) 2023-04-21 01:56:31,681 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/A in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:31,681 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/B in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:31,681 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/3363451d07214bb599a2012e5b54ffe7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/6113929947344f1d8fb2732c6f9cd59c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/c0d59d8d2fad4365b066785fbbbd2799, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/9488c294b9b04f6c87316deefe2572fe] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=102.9 K 2023-04-21 01:56:31,681 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/23eb6936fe454be486563f585572f571, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/db94cd1c7778465d97487d9c4fba9ae6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f8eefe0508194bd38c9f185e462b068f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/8b859d11d5144b8a9f743676a14ecee2] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=44.0 K 2023-04-21 01:56:31,681 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 3363451d07214bb599a2012e5b54ffe7, keycount=150, bloomtype=ROW, size=26.6 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1682042188143 2023-04-21 01:56:31,681 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 23eb6936fe454be486563f585572f571, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1682042188143 2023-04-21 01:56:31,682 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 6113929947344f1d8fb2732c6f9cd59c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1682042189238 2023-04-21 01:56:31,682 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting db94cd1c7778465d97487d9c4fba9ae6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1682042189238 2023-04-21 01:56:31,682 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting f8eefe0508194bd38c9f185e462b068f, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1682042190324 2023-04-21 01:56:31,682 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting c0d59d8d2fad4365b066785fbbbd2799, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1682042190317 2023-04-21 01:56:31,683 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 8b859d11d5144b8a9f743676a14ecee2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=884, earliestPutTs=1682042190805 2023-04-21 01:56:31,683 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 9488c294b9b04f6c87316deefe2572fe, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=884, earliestPutTs=1682042190804 2023-04-21 01:56:31,692 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#B#compaction#650 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:31,694 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#A#compaction#649 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:31,704 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/6c4d57c2a8c64c389b1c643236d3f4cb as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/6c4d57c2a8c64c389b1c643236d3f4cb 2023-04-21 01:56:31,708 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/B of 3db499d69ebc07e344a9c8f0976f4e89 into 6c4d57c2a8c64c389b1c643236d3f4cb(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:31,708 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:31,708 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/B, priority=12, startTime=1682042191679; duration=0sec 2023-04-21 01:56:31,708 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:31,708 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-21 01:56:31,709 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45016 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-21 01:56:31,709 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/C is initiating minor compaction (all files) 2023-04-21 01:56:31,709 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/C in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:31,710 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/bea9270545184044985c09a6651b500c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/06f0d56e7d7f4b51b233e38eb2d01c28, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/8c04156a1a764c3b9f9d3db60aad854b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/91b48c23cea641eeae16fc97968bbdf5] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=44.0 K 2023-04-21 01:56:31,710 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting bea9270545184044985c09a6651b500c, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1682042188143 2023-04-21 01:56:31,711 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 06f0d56e7d7f4b51b233e38eb2d01c28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1682042189238 2023-04-21 01:56:31,711 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 8c04156a1a764c3b9f9d3db60aad854b, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1682042190324 2023-04-21 01:56:31,711 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 91b48c23cea641eeae16fc97968bbdf5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=884, earliestPutTs=1682042190805 2023-04-21 01:56:31,716 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:31,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:31,720 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#C#compaction#651 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:31,722 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/d43e1789b1534e07a52489311f46c70a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/d43e1789b1534e07a52489311f46c70a 2023-04-21 01:56:31,727 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/A of 3db499d69ebc07e344a9c8f0976f4e89 into d43e1789b1534e07a52489311f46c70a(size=26.7 K), total size for store is 26.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:31,727 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:31,727 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/A, priority=12, startTime=1682042191679; duration=0sec 2023-04-21 01:56:31,727 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:31,733 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-21 01:56:31,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-21 01:56:31,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-21 01:56:31,734 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-21 01:56:31,734 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:31,734 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-21 01:56:31,734 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-21 01:56:31,735 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,735 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,745 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,745 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:31,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:31,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:31,745 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:31,745 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-21 01:56:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:31,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-21 01:56:31,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:31,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-21 01:56:31,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-21 01:56:31,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-21 01:56:31,747 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-21 01:56:31,747 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-21 01:56:31,747 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-21 01:56:31,748 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-21 01:56:31,748 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-21 01:56:31,748 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:31,748 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. started... 2023-04-21 01:56:31,748 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-21 01:56:31,748 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:31,748 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:31,748 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:31,748 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:31,748 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:31,748 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:31,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:31,932 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. as already flushing 2023-04-21 01:56:31,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042191959, max timestamp=1682042191962Num uniques 859; 2023-04-21 01:56:31,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:31,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042191959, max timestamp=1682042191962Num uniques 859; 2023-04-21 01:56:31,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682042191959, max timestamp=1682042191962Num uniques 859; 2023-04-21 01:56:31,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:31,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-21 01:56:31,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682042191980, max timestamp=1682042191991Num uniques 1018; 2023-04-21 01:56:31,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682042191980, max timestamp=1682042191991Num uniques 1018; 2023-04-21 01:56:31,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-21 01:56:31,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-21 01:56:31,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682042191980, max timestamp=1682042191991Num uniques 1018; 2023-04-21 01:56:31,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-21 01:56:32,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:32,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042252012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:32,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59372 deadline: 1682042252012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:32,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042252014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:32,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042252116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:32,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59372 deadline: 1682042252116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,120 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:32,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042252118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,130 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/e7746d040434486e810810ece77f1b72 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/e7746d040434486e810810ece77f1b72 2023-04-21 01:56:32,135 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/C of 3db499d69ebc07e344a9c8f0976f4e89 into e7746d040434486e810810ece77f1b72(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:32,135 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:32,135 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/C, priority=12, startTime=1682042191679; duration=0sec 2023-04-21 01:56:32,136 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:32,170 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421616dfa7db796458093aec5e6901aed66 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421616dfa7db796458093aec5e6901aed66 2023-04-21 01:56:32,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:32,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40131] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59372 deadline: 1682042252319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:32,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40131] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59440 deadline: 1682042252320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-21 01:56:32,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59448 deadline: 1682042252321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,575 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=895, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/366fe32ed5754d89a3188959fd7ad0c7 2023-04-21 01:56:32,590 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=895 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/4f2e9c91999a44b5a08d3d5628110ad9 2023-04-21 01:56:32,601 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=895 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/8d65e3f322184e329b0e6600282fba5a 2023-04-21 01:56:32,605 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/366fe32ed5754d89a3188959fd7ad0c7 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/366fe32ed5754d89a3188959fd7ad0c7 2023-04-21 01:56:32,612 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/366fe32ed5754d89a3188959fd7ad0c7, entries=100, sequenceid=895, filesize=18.6 K 2023-04-21 01:56:32,613 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/4f2e9c91999a44b5a08d3d5628110ad9 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/4f2e9c91999a44b5a08d3d5628110ad9 2023-04-21 01:56:32,617 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/4f2e9c91999a44b5a08d3d5628110ad9, entries=100, sequenceid=895, filesize=9.5 K 2023-04-21 01:56:32,617 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/8d65e3f322184e329b0e6600282fba5a as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/8d65e3f322184e329b0e6600282fba5a 2023-04-21 01:56:32,620 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/8d65e3f322184e329b0e6600282fba5a, entries=100, sequenceid=895, filesize=9.5 K 2023-04-21 01:56:32,621 INFO [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 3db499d69ebc07e344a9c8f0976f4e89 in 873ms, sequenceid=895, compaction requested=false 2023-04-21 01:56:32,621 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:32,621 DEBUG [rs(jenkins-hbase4.apache.org,40131,1682042019399)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:32,621 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-21 01:56:32,621 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-21 01:56:32,621 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,621 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-21 01:56:32,621 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-21 01:56:32,623 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:32,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:32,623 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:32,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:32,624 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:32,624 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-21 01:56:32,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:32,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:32,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40131] regionserver.HRegion(9158): Flush requested on 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:32,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,625 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=147.60 KB heapSize=414.61 KB 2023-04-21 01:56:32,625 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:32,625 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:32,625 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:32,625 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:32,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:32,625 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:32,625 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-21 01:56:32,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40131,1682042019399' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-21 01:56:32,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d5643b3[Count = 0] remaining members to acquire global barrier 2023-04-21 01:56:32,626 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-21 01:56:32,626 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:32,627 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:32,627 DEBUG [Thread-4060] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1cbf09b9 to 127.0.0.1:58383 2023-04-21 01:56:32,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:32,627 DEBUG [Thread-4060] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:32,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:32,627 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-21 01:56:32,627 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-21 01:56:32,627 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40131,1682042019399' in zk 2023-04-21 01:56:32,628 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,628 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-21 01:56:32,628 DEBUG [Thread-4056] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x308b3f39 to 127.0.0.1:58383 2023-04-21 01:56:32,628 DEBUG [Thread-4056] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:32,628 DEBUG [Thread-4043] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06799455 to 127.0.0.1:58383 2023-04-21 01:56:32,628 DEBUG [Thread-4043] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:32,629 DEBUG [Thread-4041] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08b57aae to 127.0.0.1:58383 2023-04-21 01:56:32,629 DEBUG [Thread-4041] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:32,630 DEBUG [Thread-4047] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f4eab6d to 127.0.0.1:58383 2023-04-21 01:56:32,630 DEBUG [Thread-4047] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:32,631 DEBUG [Thread-4054] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28164190 to 127.0.0.1:58383 2023-04-21 01:56:32,631 DEBUG [Thread-4054] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:32,632 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-21 01:56:32,632 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:32,632 DEBUG [member: 'jenkins-hbase4.apache.org,40131,1682042019399' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-21 01:56:32,632 DEBUG [Thread-4052] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x32bf16cd to 127.0.0.1:58383 2023-04-21 01:56:32,632 DEBUG [Thread-4052] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:32,632 DEBUG [Thread-4058] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x01453d93 to 127.0.0.1:58383 2023-04-21 01:56:32,632 DEBUG [Thread-4058] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:32,632 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:32,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40131,1682042019399': 2023-04-21 01:56:32,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40131,1682042019399' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-21 01:56:32,836 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-21 01:56:32,836 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-21 01:56:32,836 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-21 01:56:32,836 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-21 01:56:32,836 INFO [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-21 01:56:32,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:32,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:32,839 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:32,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-21 01:56:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-21 01:56:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-21 01:56:32,840 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-21 01:56:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,841 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-21 01:56:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-21 01:56:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,842 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,842 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:32,842 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-21 01:56:32,842 DEBUG [(jenkins-hbase4.apache.org,35449,1682042018145)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-21 01:56:32,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-21 01:56:32,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-21 01:56:32,843 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-21 01:56:32,842 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-21 01:56:32,843 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-21 01:56:32,842 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-21 01:56:32,843 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:32,843 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:32,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-21 01:56:32,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-21 01:56:32,843 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-21 01:56:32,843 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-21 01:56:32,843 DEBUG [Thread-4051] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-21 01:56:32,843 DEBUG [Thread-4051] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-21 01:56:32,943 DEBUG [Thread-4051] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-21 01:56:32,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-21 01:56:33,040 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421a3de9c23b69c4906b8f53fa856b2ba40 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421a3de9c23b69c4906b8f53fa856b2ba40 2023-04-21 01:56:33,444 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=950, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/96e629220a7c4dda8a4ec561ac0989b5 2023-04-21 01:56:33,452 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=950 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/819c26f08d204e36aeaa71aa3faf3117 2023-04-21 01:56:33,860 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=950 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/1935c009d2da4738af95a63550bf4e39 2023-04-21 01:56:33,863 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/96e629220a7c4dda8a4ec561ac0989b5 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/96e629220a7c4dda8a4ec561ac0989b5 2023-04-21 01:56:33,866 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/96e629220a7c4dda8a4ec561ac0989b5, entries=200, sequenceid=950, filesize=32.3 K 2023-04-21 01:56:33,866 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/819c26f08d204e36aeaa71aa3faf3117 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/819c26f08d204e36aeaa71aa3faf3117 2023-04-21 01:56:33,869 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/819c26f08d204e36aeaa71aa3faf3117, entries=150, sequenceid=950, filesize=11.9 K 2023-04-21 01:56:33,870 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/1935c009d2da4738af95a63550bf4e39 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/1935c009d2da4738af95a63550bf4e39 2023-04-21 01:56:33,873 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/1935c009d2da4738af95a63550bf4e39, entries=150, sequenceid=950, filesize=11.9 K 2023-04-21 01:56:33,874 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~414.56 KB/424512, currentSize=20.13 KB/20610 for 3db499d69ebc07e344a9c8f0976f4e89 in 1249ms, sequenceid=950, compaction requested=true 2023-04-21 01:56:33,874 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:33,874 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:33,874 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:56:33,874 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:33,874 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:56:33,874 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:33,875 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79458 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:56:33,875 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35442 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:56:33,875 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/A is initiating minor compaction (all files) 2023-04-21 01:56:33,875 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/B is initiating minor compaction (all files) 2023-04-21 01:56:33,875 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/A in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:33,875 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/B in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:33,875 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/d43e1789b1534e07a52489311f46c70a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/366fe32ed5754d89a3188959fd7ad0c7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/96e629220a7c4dda8a4ec561ac0989b5] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=77.6 K 2023-04-21 01:56:33,875 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/6c4d57c2a8c64c389b1c643236d3f4cb, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/4f2e9c91999a44b5a08d3d5628110ad9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/819c26f08d204e36aeaa71aa3faf3117] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=34.6 K 2023-04-21 01:56:33,875 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting d43e1789b1534e07a52489311f46c70a, keycount=150, bloomtype=ROW, size=26.7 K, encoding=NONE, compression=NONE, seqNum=884, earliestPutTs=1682042190805 2023-04-21 01:56:33,875 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 6c4d57c2a8c64c389b1c643236d3f4cb, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=884, earliestPutTs=1682042190805 2023-04-21 01:56:33,876 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 366fe32ed5754d89a3188959fd7ad0c7, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=895, earliestPutTs=1682042190817 2023-04-21 01:56:33,876 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 4f2e9c91999a44b5a08d3d5628110ad9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=895, earliestPutTs=1682042190817 2023-04-21 01:56:33,876 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] compactions.Compactor(207): Compacting 96e629220a7c4dda8a4ec561ac0989b5, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=950, earliestPutTs=1682042192008 2023-04-21 01:56:33,876 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 819c26f08d204e36aeaa71aa3faf3117, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=950, earliestPutTs=1682042192008 2023-04-21 01:56:33,881 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#B#compaction#658 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:33,893 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#A#compaction#659 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:34,293 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/bdcdf00b5a08464598f4fbcae498cbe5 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/bdcdf00b5a08464598f4fbcae498cbe5 2023-04-21 01:56:34,297 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/B of 3db499d69ebc07e344a9c8f0976f4e89 into bdcdf00b5a08464598f4fbcae498cbe5(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:34,297 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:34,297 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/B, priority=13, startTime=1682042193874; duration=0sec 2023-04-21 01:56:34,297 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-21 01:56:34,297 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-21 01:56:34,298 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35408 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-21 01:56:34,298 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1912): 3db499d69ebc07e344a9c8f0976f4e89/C is initiating minor compaction (all files) 2023-04-21 01:56:34,298 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3db499d69ebc07e344a9c8f0976f4e89/C in TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:34,298 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/e7746d040434486e810810ece77f1b72, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/8d65e3f322184e329b0e6600282fba5a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/1935c009d2da4738af95a63550bf4e39] into tmpdir=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp, totalSize=34.6 K 2023-04-21 01:56:34,299 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting e7746d040434486e810810ece77f1b72, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=884, earliestPutTs=1682042190805 2023-04-21 01:56:34,299 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 8d65e3f322184e329b0e6600282fba5a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=895, earliestPutTs=1682042190817 2023-04-21 01:56:34,299 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] compactions.Compactor(207): Compacting 1935c009d2da4738af95a63550bf4e39, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=950, earliestPutTs=1682042192008 2023-04-21 01:56:34,304 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] throttle.PressureAwareThroughputController(145): 3db499d69ebc07e344a9c8f0976f4e89#C#compaction#660 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-21 01:56:34,711 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/a0599347166a4afaa23a3856c280e3cf as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/a0599347166a4afaa23a3856c280e3cf 2023-04-21 01:56:34,715 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/C of 3db499d69ebc07e344a9c8f0976f4e89 into a0599347166a4afaa23a3856c280e3cf(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-21 01:56:34,715 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:34,715 INFO [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/C, priority=13, startTime=1682042193874; duration=0sec 2023-04-21 01:56:34,715 DEBUG [RS:0;jenkins-hbase4:40131-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:35,108 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/a859dc1efef8439aa050156c7b20cc57 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/a859dc1efef8439aa050156c7b20cc57 2023-04-21 01:56:35,112 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3db499d69ebc07e344a9c8f0976f4e89/A of 3db499d69ebc07e344a9c8f0976f4e89 into a859dc1efef8439aa050156c7b20cc57(size=26.8 K), total size for store is 26.8 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-21 01:56:35,112 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:35,112 INFO [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89., storeName=3db499d69ebc07e344a9c8f0976f4e89/A, priority=13, startTime=1682042193874; duration=1sec 2023-04-21 01:56:35,112 DEBUG [RS:0;jenkins-hbase4:40131-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-21 01:56:35,273 DEBUG [Thread-4049] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07d997c6 to 127.0.0.1:58383 2023-04-21 01:56:35,273 DEBUG [Thread-4049] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:39,497 DEBUG [Thread-4045] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66c25427 to 127.0.0.1:58383 2023-04-21 01:56:39,497 DEBUG [Thread-4045] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:39,497 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-21 01:56:39,497 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 271 2023-04-21 01:56:39,498 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 239 2023-04-21 01:56:39,498 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 113 2023-04-21 01:56:39,498 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 144 2023-04-21 01:56:39,498 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(394): wrote 58 2023-04-21 01:56:39,498 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-21 01:56:39,498 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(398): read 4381 2023-04-21 01:56:39,498 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(398): read 4358 2023-04-21 01:56:39,498 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(398): read 4344 2023-04-21 01:56:39,498 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(398): read 4375 2023-04-21 01:56:39,498 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(398): read 4411 2023-04-21 01:56:39,498 INFO [Listener at localhost/39939] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-21 01:56:39,498 INFO [Listener at localhost/39939] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-21 01:56:39,498 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4db22237 to 127.0.0.1:58383 2023-04-21 01:56:39,498 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:39,499 INFO [Listener at localhost/39939] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-21 01:56:39,499 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-21 01:56:39,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-21 01:56:39,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-21 01:56:39,502 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042199502"}]},"ts":"1682042199502"} 2023-04-21 01:56:39,503 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-21 01:56:39,504 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-21 01:56:39,505 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3db499d69ebc07e344a9c8f0976f4e89, UNASSIGN}] 2023-04-21 01:56:39,506 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3db499d69ebc07e344a9c8f0976f4e89, UNASSIGN 2023-04-21 01:56:39,507 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=3db499d69ebc07e344a9c8f0976f4e89, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:39,507 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042199507"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682042199507"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682042199507"}]},"ts":"1682042199507"} 2023-04-21 01:56:39,508 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399}] 2023-04-21 01:56:39,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-21 01:56:39,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:39,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 3db499d69ebc07e344a9c8f0976f4e89, disabling compactions & flushes 2023-04-21 01:56:39,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:39,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:39,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. after waiting 0 ms 2023-04-21 01:56:39,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:39,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 3db499d69ebc07e344a9c8f0976f4e89 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-21 01:56:39,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=A 2023-04-21 01:56:39,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:39,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=B 2023-04-21 01:56:39,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:39,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3db499d69ebc07e344a9c8f0976f4e89, store=C 2023-04-21 01:56:39,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-21 01:56:39,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-21 01:56:40,071 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230421bbba56bcfe4e4e8d87ced36267a83c7b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421bbba56bcfe4e4e8d87ced36267a83c7b 2023-04-21 01:56:40,075 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=961, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/a93115770e3e40d7aa43cef05ddd7edd 2023-04-21 01:56:40,081 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=961 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/56dd818c3a174190a614f33fcdaade98 2023-04-21 01:56:40,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-21 01:56:40,488 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=961 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/96b6f7635770445dbef03ca35969867f 2023-04-21 01:56:40,491 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/A/a93115770e3e40d7aa43cef05ddd7edd as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/a93115770e3e40d7aa43cef05ddd7edd 2023-04-21 01:56:40,494 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/a93115770e3e40d7aa43cef05ddd7edd, entries=150, sequenceid=961, filesize=25.4 K 2023-04-21 01:56:40,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/B/56dd818c3a174190a614f33fcdaade98 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/56dd818c3a174190a614f33fcdaade98 2023-04-21 01:56:40,498 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/56dd818c3a174190a614f33fcdaade98, entries=150, sequenceid=961, filesize=11.9 K 2023-04-21 01:56:40,498 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/.tmp/C/96b6f7635770445dbef03ca35969867f as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/96b6f7635770445dbef03ca35969867f 2023-04-21 01:56:40,501 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/96b6f7635770445dbef03ca35969867f, entries=150, sequenceid=961, filesize=11.9 K 2023-04-21 01:56:40,501 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 3db499d69ebc07e344a9c8f0976f4e89 in 842ms, sequenceid=961, compaction requested=false 2023-04-21 01:56:40,503 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/2136a8a0c3d44f65bc38e86695ae8be6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/34a17469e14c4b0c82e1b0ae156c3a47, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/f3785e1b685b4e5e9b3abdde7f023aea, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/88484729d9ec4ce9a507c59660d63a60, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/c7c5e5dc96bc43f69c302ccd8b08f43c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/d374075ee3084089b8a6733b0ccb7868, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/9950c940a6bf4605bb8658dda72447e8, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/2820906420184a35926a6c4ef1185a4a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/eae8913ba7db4ef0a06d3656d1e844c6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/51536db93ee444e9a51b3f8a5751b0fa, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/da6d97a6aebd4a9f94c11ad293da892f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/1332d07f86c840f5af8d87d5a354c320, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/ff6b8c297b864c2195cf7166b031ed79, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/fa445f57591b4e73864515e4a84b14be, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/54b9cab515c84816abfb7ec5d7bf2c31, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/6bf9eb90b83c409fac804e8b9ed93be6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/2451f2f328bc4e04aa20c602824b75d5, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/e2c5f20a9023466bb9746546006a1af0, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/0bd52b7b38c24018bac400d0ea5e7214, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/e9d046a803a14dda96d1ca1c7a351814, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/64bc65738c5448fba614c10f771d31dc, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/ee5ce701517446af9ffad29d3cac5138, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/4ce5468978e143c4aee82a30b92a0db4, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/b6ad7c5a5c804e4ca1b79140936b9d81, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/bc865a4c5cde448ba3293d74ca313086, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/e23bf632e3aa4c898586f4e7f713eba9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/aa48e7582b934dc4a1b9996c45f07196, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/d3129a8a9c9a4970a2b771b3c64b1d03, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/fd7040044eff43f3a7007256de9a9964, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/65bcdff729d74de882c7f6d2015f44bd, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/251b64bcdfba45f7b87c73634e2f4229, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/9f3ac2aea3fa4cfeb31a2b098994f740, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/7e63469621814a87ad3fe3cac6a26e79, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/dfe27bbf51ac4c7eb4bf0e853133b81e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/a579fd93cd7c42489b5aaf437b5a1791, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/3363451d07214bb599a2012e5b54ffe7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/6113929947344f1d8fb2732c6f9cd59c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/c0d59d8d2fad4365b066785fbbbd2799, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/9488c294b9b04f6c87316deefe2572fe, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/d43e1789b1534e07a52489311f46c70a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/366fe32ed5754d89a3188959fd7ad0c7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/96e629220a7c4dda8a4ec561ac0989b5] to archive 2023-04-21 01:56:40,503 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-21 01:56:40,505 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/2136a8a0c3d44f65bc38e86695ae8be6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/2136a8a0c3d44f65bc38e86695ae8be6 2023-04-21 01:56:40,506 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/34a17469e14c4b0c82e1b0ae156c3a47 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/34a17469e14c4b0c82e1b0ae156c3a47 2023-04-21 01:56:40,506 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/f3785e1b685b4e5e9b3abdde7f023aea to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/f3785e1b685b4e5e9b3abdde7f023aea 2023-04-21 01:56:40,507 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/88484729d9ec4ce9a507c59660d63a60 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/88484729d9ec4ce9a507c59660d63a60 2023-04-21 01:56:40,508 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/c7c5e5dc96bc43f69c302ccd8b08f43c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/c7c5e5dc96bc43f69c302ccd8b08f43c 2023-04-21 01:56:40,509 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/d374075ee3084089b8a6733b0ccb7868 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/d374075ee3084089b8a6733b0ccb7868 2023-04-21 01:56:40,510 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/9950c940a6bf4605bb8658dda72447e8 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/9950c940a6bf4605bb8658dda72447e8 2023-04-21 01:56:40,510 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/2820906420184a35926a6c4ef1185a4a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/2820906420184a35926a6c4ef1185a4a 2023-04-21 01:56:40,511 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/eae8913ba7db4ef0a06d3656d1e844c6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/eae8913ba7db4ef0a06d3656d1e844c6 2023-04-21 01:56:40,512 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/51536db93ee444e9a51b3f8a5751b0fa to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/51536db93ee444e9a51b3f8a5751b0fa 2023-04-21 01:56:40,513 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/da6d97a6aebd4a9f94c11ad293da892f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/da6d97a6aebd4a9f94c11ad293da892f 2023-04-21 01:56:40,513 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/1332d07f86c840f5af8d87d5a354c320 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/1332d07f86c840f5af8d87d5a354c320 2023-04-21 01:56:40,514 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/ff6b8c297b864c2195cf7166b031ed79 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/ff6b8c297b864c2195cf7166b031ed79 2023-04-21 01:56:40,515 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/fa445f57591b4e73864515e4a84b14be to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/fa445f57591b4e73864515e4a84b14be 2023-04-21 01:56:40,516 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/54b9cab515c84816abfb7ec5d7bf2c31 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/54b9cab515c84816abfb7ec5d7bf2c31 2023-04-21 01:56:40,516 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/6bf9eb90b83c409fac804e8b9ed93be6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/6bf9eb90b83c409fac804e8b9ed93be6 2023-04-21 01:56:40,517 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/2451f2f328bc4e04aa20c602824b75d5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/2451f2f328bc4e04aa20c602824b75d5 2023-04-21 01:56:40,518 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/e2c5f20a9023466bb9746546006a1af0 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/e2c5f20a9023466bb9746546006a1af0 2023-04-21 01:56:40,519 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/0bd52b7b38c24018bac400d0ea5e7214 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/0bd52b7b38c24018bac400d0ea5e7214 2023-04-21 01:56:40,519 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/e9d046a803a14dda96d1ca1c7a351814 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/e9d046a803a14dda96d1ca1c7a351814 2023-04-21 01:56:40,520 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/64bc65738c5448fba614c10f771d31dc to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/64bc65738c5448fba614c10f771d31dc 2023-04-21 01:56:40,521 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/ee5ce701517446af9ffad29d3cac5138 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/ee5ce701517446af9ffad29d3cac5138 2023-04-21 01:56:40,522 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/4ce5468978e143c4aee82a30b92a0db4 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/4ce5468978e143c4aee82a30b92a0db4 2023-04-21 01:56:40,522 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/b6ad7c5a5c804e4ca1b79140936b9d81 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/b6ad7c5a5c804e4ca1b79140936b9d81 2023-04-21 01:56:40,523 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/bc865a4c5cde448ba3293d74ca313086 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/bc865a4c5cde448ba3293d74ca313086 2023-04-21 01:56:40,524 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/e23bf632e3aa4c898586f4e7f713eba9 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/e23bf632e3aa4c898586f4e7f713eba9 2023-04-21 01:56:40,525 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/aa48e7582b934dc4a1b9996c45f07196 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/aa48e7582b934dc4a1b9996c45f07196 2023-04-21 01:56:40,526 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/d3129a8a9c9a4970a2b771b3c64b1d03 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/d3129a8a9c9a4970a2b771b3c64b1d03 2023-04-21 01:56:40,526 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/fd7040044eff43f3a7007256de9a9964 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/fd7040044eff43f3a7007256de9a9964 2023-04-21 01:56:40,527 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/65bcdff729d74de882c7f6d2015f44bd to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/65bcdff729d74de882c7f6d2015f44bd 2023-04-21 01:56:40,528 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/251b64bcdfba45f7b87c73634e2f4229 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/251b64bcdfba45f7b87c73634e2f4229 2023-04-21 01:56:40,529 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/9f3ac2aea3fa4cfeb31a2b098994f740 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/9f3ac2aea3fa4cfeb31a2b098994f740 2023-04-21 01:56:40,529 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/7e63469621814a87ad3fe3cac6a26e79 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/7e63469621814a87ad3fe3cac6a26e79 2023-04-21 01:56:40,530 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/dfe27bbf51ac4c7eb4bf0e853133b81e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/dfe27bbf51ac4c7eb4bf0e853133b81e 2023-04-21 01:56:40,531 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/a579fd93cd7c42489b5aaf437b5a1791 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/a579fd93cd7c42489b5aaf437b5a1791 2023-04-21 01:56:40,532 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/3363451d07214bb599a2012e5b54ffe7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/3363451d07214bb599a2012e5b54ffe7 2023-04-21 01:56:40,532 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/6113929947344f1d8fb2732c6f9cd59c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/6113929947344f1d8fb2732c6f9cd59c 2023-04-21 01:56:40,533 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/c0d59d8d2fad4365b066785fbbbd2799 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/c0d59d8d2fad4365b066785fbbbd2799 2023-04-21 01:56:40,534 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/9488c294b9b04f6c87316deefe2572fe to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/9488c294b9b04f6c87316deefe2572fe 2023-04-21 01:56:40,535 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/d43e1789b1534e07a52489311f46c70a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/d43e1789b1534e07a52489311f46c70a 2023-04-21 01:56:40,536 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/366fe32ed5754d89a3188959fd7ad0c7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/366fe32ed5754d89a3188959fd7ad0c7 2023-04-21 01:56:40,536 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/96e629220a7c4dda8a4ec561ac0989b5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/96e629220a7c4dda8a4ec561ac0989b5 2023-04-21 01:56:40,539 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f2fe649d59804d4081208a58f0acaf84, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/0a7825559152432fba94fa5806721d6e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/2876bbc1114d4bbf9fa33300143c0db9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/88aee48cdfd84ad6892ca1c284da111e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b05289a8617d4fd3b179dea8ba4a2964, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b6553f2485594a2c9d66fa3a17f32af4, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/6cea4d6900984fc39da87ff38dbb43ef, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/3aadde79b4044f91abbca813c788b998, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f5e94ad539eb4438ba013adbfacebf35, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/7b6ebf979a4b432599c0c572d5b5d5bf, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/336e9840de8d4df5b5f76cfa03702d7a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/3d706b79672c408fba1bf965a759c8d1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/5e834f19a0a044409ef9323f767103d0, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/94becbef18394a3b87d0dd540f1f2b96, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f409ecaaa7b34014a59b52b9e7701d95, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/cab2d87a194845a18ecc39c7ec9f99e7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/d3da22339d974f71a2f694fc4457135f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/e0e875c40ec249c18fc83fe8b92e42c9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/4a0ee5b83ecd4f5eac86512eeca704ff, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/8fe2f804b904495e8a384d75a4b91bb0, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f26fa546fb354ebd931192538dea42b7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/c550238701a149ecafb2d633e4c029c9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b886f51645c64088985207d4b2b4de47, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/c210c198f562487dbaf2cf710e01f118, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/dc70ef6de0364f5c942af3818c0feac1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/8763dbe684b74fdba80bb8dbfc6e07b6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/8d88c0739b66420e862def9da551c0e2, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b75a9dece28c458b93e00081b0f7d080, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/63a91fa2e82242f3a18e9017e736f0d6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/07004024efd44692bcacc62889e20de3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/356449ebf565424f92e69191defd18e3, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/79c3b366832c4062a76fa97f0394396c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/c2032ef0742d4fe086df3db9b9d1257e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b85c0503e0524af6a22827f937241b04, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/23eb6936fe454be486563f585572f571, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/07e855a60d3c4800b3f4c12377b1c2d7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/db94cd1c7778465d97487d9c4fba9ae6, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f8eefe0508194bd38c9f185e462b068f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/6c4d57c2a8c64c389b1c643236d3f4cb, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/8b859d11d5144b8a9f743676a14ecee2, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/4f2e9c91999a44b5a08d3d5628110ad9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/819c26f08d204e36aeaa71aa3faf3117] to archive 2023-04-21 01:56:40,540 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-21 01:56:40,541 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f2fe649d59804d4081208a58f0acaf84 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f2fe649d59804d4081208a58f0acaf84 2023-04-21 01:56:40,542 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/0a7825559152432fba94fa5806721d6e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/0a7825559152432fba94fa5806721d6e 2023-04-21 01:56:40,543 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/2876bbc1114d4bbf9fa33300143c0db9 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/2876bbc1114d4bbf9fa33300143c0db9 2023-04-21 01:56:40,544 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/88aee48cdfd84ad6892ca1c284da111e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/88aee48cdfd84ad6892ca1c284da111e 2023-04-21 01:56:40,545 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b05289a8617d4fd3b179dea8ba4a2964 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b05289a8617d4fd3b179dea8ba4a2964 2023-04-21 01:56:40,545 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b6553f2485594a2c9d66fa3a17f32af4 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b6553f2485594a2c9d66fa3a17f32af4 2023-04-21 01:56:40,546 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/6cea4d6900984fc39da87ff38dbb43ef to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/6cea4d6900984fc39da87ff38dbb43ef 2023-04-21 01:56:40,547 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/3aadde79b4044f91abbca813c788b998 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/3aadde79b4044f91abbca813c788b998 2023-04-21 01:56:40,548 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f5e94ad539eb4438ba013adbfacebf35 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f5e94ad539eb4438ba013adbfacebf35 2023-04-21 01:56:40,549 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/7b6ebf979a4b432599c0c572d5b5d5bf to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/7b6ebf979a4b432599c0c572d5b5d5bf 2023-04-21 01:56:40,549 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/336e9840de8d4df5b5f76cfa03702d7a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/336e9840de8d4df5b5f76cfa03702d7a 2023-04-21 01:56:40,550 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/3d706b79672c408fba1bf965a759c8d1 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/3d706b79672c408fba1bf965a759c8d1 2023-04-21 01:56:40,551 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/5e834f19a0a044409ef9323f767103d0 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/5e834f19a0a044409ef9323f767103d0 2023-04-21 01:56:40,552 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/94becbef18394a3b87d0dd540f1f2b96 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/94becbef18394a3b87d0dd540f1f2b96 2023-04-21 01:56:40,553 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f409ecaaa7b34014a59b52b9e7701d95 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f409ecaaa7b34014a59b52b9e7701d95 2023-04-21 01:56:40,553 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/cab2d87a194845a18ecc39c7ec9f99e7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/cab2d87a194845a18ecc39c7ec9f99e7 2023-04-21 01:56:40,554 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/d3da22339d974f71a2f694fc4457135f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/d3da22339d974f71a2f694fc4457135f 2023-04-21 01:56:40,555 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/e0e875c40ec249c18fc83fe8b92e42c9 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/e0e875c40ec249c18fc83fe8b92e42c9 2023-04-21 01:56:40,556 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/4a0ee5b83ecd4f5eac86512eeca704ff to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/4a0ee5b83ecd4f5eac86512eeca704ff 2023-04-21 01:56:40,557 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/8fe2f804b904495e8a384d75a4b91bb0 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/8fe2f804b904495e8a384d75a4b91bb0 2023-04-21 01:56:40,558 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f26fa546fb354ebd931192538dea42b7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f26fa546fb354ebd931192538dea42b7 2023-04-21 01:56:40,559 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/c550238701a149ecafb2d633e4c029c9 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/c550238701a149ecafb2d633e4c029c9 2023-04-21 01:56:40,560 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b886f51645c64088985207d4b2b4de47 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b886f51645c64088985207d4b2b4de47 2023-04-21 01:56:40,561 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/c210c198f562487dbaf2cf710e01f118 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/c210c198f562487dbaf2cf710e01f118 2023-04-21 01:56:40,562 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/dc70ef6de0364f5c942af3818c0feac1 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/dc70ef6de0364f5c942af3818c0feac1 2023-04-21 01:56:40,563 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/8763dbe684b74fdba80bb8dbfc6e07b6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/8763dbe684b74fdba80bb8dbfc6e07b6 2023-04-21 01:56:40,563 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/8d88c0739b66420e862def9da551c0e2 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/8d88c0739b66420e862def9da551c0e2 2023-04-21 01:56:40,564 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b75a9dece28c458b93e00081b0f7d080 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b75a9dece28c458b93e00081b0f7d080 2023-04-21 01:56:40,565 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/63a91fa2e82242f3a18e9017e736f0d6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/63a91fa2e82242f3a18e9017e736f0d6 2023-04-21 01:56:40,566 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/07004024efd44692bcacc62889e20de3 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/07004024efd44692bcacc62889e20de3 2023-04-21 01:56:40,567 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/356449ebf565424f92e69191defd18e3 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/356449ebf565424f92e69191defd18e3 2023-04-21 01:56:40,568 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/79c3b366832c4062a76fa97f0394396c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/79c3b366832c4062a76fa97f0394396c 2023-04-21 01:56:40,569 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/c2032ef0742d4fe086df3db9b9d1257e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/c2032ef0742d4fe086df3db9b9d1257e 2023-04-21 01:56:40,570 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b85c0503e0524af6a22827f937241b04 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/b85c0503e0524af6a22827f937241b04 2023-04-21 01:56:40,571 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/23eb6936fe454be486563f585572f571 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/23eb6936fe454be486563f585572f571 2023-04-21 01:56:40,572 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/07e855a60d3c4800b3f4c12377b1c2d7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/07e855a60d3c4800b3f4c12377b1c2d7 2023-04-21 01:56:40,573 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/db94cd1c7778465d97487d9c4fba9ae6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/db94cd1c7778465d97487d9c4fba9ae6 2023-04-21 01:56:40,574 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f8eefe0508194bd38c9f185e462b068f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/f8eefe0508194bd38c9f185e462b068f 2023-04-21 01:56:40,575 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/6c4d57c2a8c64c389b1c643236d3f4cb to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/6c4d57c2a8c64c389b1c643236d3f4cb 2023-04-21 01:56:40,576 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/8b859d11d5144b8a9f743676a14ecee2 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/8b859d11d5144b8a9f743676a14ecee2 2023-04-21 01:56:40,577 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/4f2e9c91999a44b5a08d3d5628110ad9 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/4f2e9c91999a44b5a08d3d5628110ad9 2023-04-21 01:56:40,578 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/819c26f08d204e36aeaa71aa3faf3117 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/819c26f08d204e36aeaa71aa3faf3117 2023-04-21 01:56:40,581 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/36cdc0e7440f4c6a83ba597747e6f621, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/bef9d3a04bce490480053b7ae8aa66f5, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/3da7a0fb83064751b5a85f3912b70adf, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/a5402c1baa19407ea0cb4634ec74aa1f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/e444ac010dfa43508735285f7852a9ea, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/64d25ca4b84d41fdb00981cc29d87ebe, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/12e49110ccca498b908c5e24b2c2c597, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/c471342b3c4f46a78addb7e03967bd9f, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/d3a13ace148e4b9eb477b3aae8a94f13, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/3da8657a573d4f2998277211250ba391, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/66aeca9ed0b949efa8d36c7e6ec7c7fe, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/a50cd2ca99ec4595960a3447a636de3c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/edfcb40378ca40ec89731390c533ec45, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/beea4d279260456fbe82de606664c3c7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/4fcf6fc82a494016a0378408609aa54b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/88a7c3afdcc84c969f1ab61e5c2afc13, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/eacd4847f1ac49cb9b3ebccea3dc9a02, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/e2c3440699bb4ac6ba873e5debdb251c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/518bdabcb6b24ef5b5e91e4400b69d70, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/8a5c32de03a2431a8ebee09a2950b12c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/cb3db025ce5d4866a241a08d1f4bc071, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/831cf1cd13d346139c347af578161076, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/12a8f91ba9ac4edbb0dabbfa4d3b0a8e, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/613effdeb7e64249a433e37d58011ce9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/653c05bf8ec34e6db7d1c5a4a76be14a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/c513bf031c8944778da2938f71f4fbd1, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/9083e003a7fd4ba2ae67fc6969575cb7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/eafaeda6fb07432ebd476c0d066f2ec9, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/d96b4358870f4606bcd24e350bdc4e52, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/547e25477c37476094b37a9c80795ae7, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/a25b5161b381478cab64040638fc9382, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/eeb560185b83484287edcafcd41038ca, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/1b63b346cf124c4cb878a1c3f3358337, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/bea9270545184044985c09a6651b500c, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/28213a91e4544bc59712ad1e340ac180, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/06f0d56e7d7f4b51b233e38eb2d01c28, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/8c04156a1a764c3b9f9d3db60aad854b, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/e7746d040434486e810810ece77f1b72, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/91b48c23cea641eeae16fc97968bbdf5, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/8d65e3f322184e329b0e6600282fba5a, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/1935c009d2da4738af95a63550bf4e39] to archive 2023-04-21 01:56:40,582 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-21 01:56:40,583 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/36cdc0e7440f4c6a83ba597747e6f621 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/36cdc0e7440f4c6a83ba597747e6f621 2023-04-21 01:56:40,585 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/bef9d3a04bce490480053b7ae8aa66f5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/bef9d3a04bce490480053b7ae8aa66f5 2023-04-21 01:56:40,586 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/3da7a0fb83064751b5a85f3912b70adf to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/3da7a0fb83064751b5a85f3912b70adf 2023-04-21 01:56:40,587 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/a5402c1baa19407ea0cb4634ec74aa1f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/a5402c1baa19407ea0cb4634ec74aa1f 2023-04-21 01:56:40,588 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/e444ac010dfa43508735285f7852a9ea to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/e444ac010dfa43508735285f7852a9ea 2023-04-21 01:56:40,589 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/64d25ca4b84d41fdb00981cc29d87ebe to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/64d25ca4b84d41fdb00981cc29d87ebe 2023-04-21 01:56:40,590 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/12e49110ccca498b908c5e24b2c2c597 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/12e49110ccca498b908c5e24b2c2c597 2023-04-21 01:56:40,591 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/c471342b3c4f46a78addb7e03967bd9f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/c471342b3c4f46a78addb7e03967bd9f 2023-04-21 01:56:40,592 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/d3a13ace148e4b9eb477b3aae8a94f13 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/d3a13ace148e4b9eb477b3aae8a94f13 2023-04-21 01:56:40,593 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/3da8657a573d4f2998277211250ba391 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/3da8657a573d4f2998277211250ba391 2023-04-21 01:56:40,594 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/66aeca9ed0b949efa8d36c7e6ec7c7fe to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/66aeca9ed0b949efa8d36c7e6ec7c7fe 2023-04-21 01:56:40,595 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/a50cd2ca99ec4595960a3447a636de3c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/a50cd2ca99ec4595960a3447a636de3c 2023-04-21 01:56:40,596 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/edfcb40378ca40ec89731390c533ec45 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/edfcb40378ca40ec89731390c533ec45 2023-04-21 01:56:40,597 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/beea4d279260456fbe82de606664c3c7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/beea4d279260456fbe82de606664c3c7 2023-04-21 01:56:40,598 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/4fcf6fc82a494016a0378408609aa54b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/4fcf6fc82a494016a0378408609aa54b 2023-04-21 01:56:40,599 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/88a7c3afdcc84c969f1ab61e5c2afc13 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/88a7c3afdcc84c969f1ab61e5c2afc13 2023-04-21 01:56:40,601 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/eacd4847f1ac49cb9b3ebccea3dc9a02 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/eacd4847f1ac49cb9b3ebccea3dc9a02 2023-04-21 01:56:40,602 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/e2c3440699bb4ac6ba873e5debdb251c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/e2c3440699bb4ac6ba873e5debdb251c 2023-04-21 01:56:40,603 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/518bdabcb6b24ef5b5e91e4400b69d70 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/518bdabcb6b24ef5b5e91e4400b69d70 2023-04-21 01:56:40,604 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/8a5c32de03a2431a8ebee09a2950b12c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/8a5c32de03a2431a8ebee09a2950b12c 2023-04-21 01:56:40,605 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/cb3db025ce5d4866a241a08d1f4bc071 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/cb3db025ce5d4866a241a08d1f4bc071 2023-04-21 01:56:40,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-21 01:56:40,605 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/831cf1cd13d346139c347af578161076 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/831cf1cd13d346139c347af578161076 2023-04-21 01:56:40,606 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/12a8f91ba9ac4edbb0dabbfa4d3b0a8e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/12a8f91ba9ac4edbb0dabbfa4d3b0a8e 2023-04-21 01:56:40,607 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/613effdeb7e64249a433e37d58011ce9 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/613effdeb7e64249a433e37d58011ce9 2023-04-21 01:56:40,608 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/653c05bf8ec34e6db7d1c5a4a76be14a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/653c05bf8ec34e6db7d1c5a4a76be14a 2023-04-21 01:56:40,609 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/c513bf031c8944778da2938f71f4fbd1 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/c513bf031c8944778da2938f71f4fbd1 2023-04-21 01:56:40,610 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/9083e003a7fd4ba2ae67fc6969575cb7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/9083e003a7fd4ba2ae67fc6969575cb7 2023-04-21 01:56:40,611 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/eafaeda6fb07432ebd476c0d066f2ec9 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/eafaeda6fb07432ebd476c0d066f2ec9 2023-04-21 01:56:40,611 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/d96b4358870f4606bcd24e350bdc4e52 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/d96b4358870f4606bcd24e350bdc4e52 2023-04-21 01:56:40,612 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/547e25477c37476094b37a9c80795ae7 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/547e25477c37476094b37a9c80795ae7 2023-04-21 01:56:40,613 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/a25b5161b381478cab64040638fc9382 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/a25b5161b381478cab64040638fc9382 2023-04-21 01:56:40,614 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/eeb560185b83484287edcafcd41038ca to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/eeb560185b83484287edcafcd41038ca 2023-04-21 01:56:40,615 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/1b63b346cf124c4cb878a1c3f3358337 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/1b63b346cf124c4cb878a1c3f3358337 2023-04-21 01:56:40,616 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/bea9270545184044985c09a6651b500c to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/bea9270545184044985c09a6651b500c 2023-04-21 01:56:40,617 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/28213a91e4544bc59712ad1e340ac180 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/28213a91e4544bc59712ad1e340ac180 2023-04-21 01:56:40,617 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/06f0d56e7d7f4b51b233e38eb2d01c28 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/06f0d56e7d7f4b51b233e38eb2d01c28 2023-04-21 01:56:40,618 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/8c04156a1a764c3b9f9d3db60aad854b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/8c04156a1a764c3b9f9d3db60aad854b 2023-04-21 01:56:40,619 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/e7746d040434486e810810ece77f1b72 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/e7746d040434486e810810ece77f1b72 2023-04-21 01:56:40,620 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/91b48c23cea641eeae16fc97968bbdf5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/91b48c23cea641eeae16fc97968bbdf5 2023-04-21 01:56:40,621 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/8d65e3f322184e329b0e6600282fba5a to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/8d65e3f322184e329b0e6600282fba5a 2023-04-21 01:56:40,622 DEBUG [StoreCloser-TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/1935c009d2da4738af95a63550bf4e39 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/1935c009d2da4738af95a63550bf4e39 2023-04-21 01:56:40,625 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/recovered.edits/964.seqid, newMaxSeqId=964, maxSeqId=4 2023-04-21 01:56:40,626 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89. 2023-04-21 01:56:40,627 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 3db499d69ebc07e344a9c8f0976f4e89: 2023-04-21 01:56:40,628 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:40,628 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=3db499d69ebc07e344a9c8f0976f4e89, regionState=CLOSED 2023-04-21 01:56:40,628 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682042200628"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682042200628"}]},"ts":"1682042200628"} 2023-04-21 01:56:40,630 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-21 01:56:40,630 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 3db499d69ebc07e344a9c8f0976f4e89, server=jenkins-hbase4.apache.org,40131,1682042019399 in 1.1210 sec 2023-04-21 01:56:40,631 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-21 01:56:40,631 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3db499d69ebc07e344a9c8f0976f4e89, UNASSIGN in 1.1250 sec 2023-04-21 01:56:40,632 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682042200632"}]},"ts":"1682042200632"} 2023-04-21 01:56:40,632 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-21 01:56:40,634 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-21 01:56:40,636 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1360 sec 2023-04-21 01:56:40,697 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-21 01:56:41,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-21 01:56:41,606 INFO [Listener at localhost/39939] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-21 01:56:41,607 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-21 01:56:41,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:56:41,609 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:56:41,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-21 01:56:41,610 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:56:41,613 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:41,615 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A, FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B, FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C, FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/recovered.edits] 2023-04-21 01:56:41,617 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/a859dc1efef8439aa050156c7b20cc57 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/a859dc1efef8439aa050156c7b20cc57 2023-04-21 01:56:41,618 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/a93115770e3e40d7aa43cef05ddd7edd to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/A/a93115770e3e40d7aa43cef05ddd7edd 2023-04-21 01:56:41,620 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/56dd818c3a174190a614f33fcdaade98 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/56dd818c3a174190a614f33fcdaade98 2023-04-21 01:56:41,621 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/bdcdf00b5a08464598f4fbcae498cbe5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/B/bdcdf00b5a08464598f4fbcae498cbe5 2023-04-21 01:56:41,623 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/96b6f7635770445dbef03ca35969867f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/96b6f7635770445dbef03ca35969867f 2023-04-21 01:56:41,624 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/a0599347166a4afaa23a3856c280e3cf to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/C/a0599347166a4afaa23a3856c280e3cf 2023-04-21 01:56:41,626 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/recovered.edits/964.seqid to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89/recovered.edits/964.seqid 2023-04-21 01:56:41,627 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/.tmp/data/default/TestAcidGuarantees/3db499d69ebc07e344a9c8f0976f4e89 2023-04-21 01:56:41,627 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-21 01:56:41,627 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-21 01:56:41,628 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-21 01:56:41,630 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421006b1b78357b443a86d1c270293f09d2 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421006b1b78357b443a86d1c270293f09d2 2023-04-21 01:56:41,631 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304210093fe56cdf04f1d85ea3ffa60c403ea to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304210093fe56cdf04f1d85ea3ffa60c403ea 2023-04-21 01:56:41,632 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304210808fa8d28a043c89631870be426e623 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304210808fa8d28a043c89631870be426e623 2023-04-21 01:56:41,633 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304211a205c12e6be4065b2f94c14c6b49ab5 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304211a205c12e6be4065b2f94c14c6b49ab5 2023-04-21 01:56:41,634 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304211e8531ca41a5475db41c7323f681fe80 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304211e8531ca41a5475db41c7323f681fe80 2023-04-21 01:56:41,635 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304212147536e57444d408ec2b309782573cd to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304212147536e57444d408ec2b309782573cd 2023-04-21 01:56:41,635 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042122e828da8fdf444eb1c00dae411c2c39 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042122e828da8fdf444eb1c00dae411c2c39 2023-04-21 01:56:41,636 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042134eb0ea03d1c44fe9665eb052c3f0f2d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042134eb0ea03d1c44fe9665eb052c3f0f2d 2023-04-21 01:56:41,637 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304213546ce539b374bbdb7101e43ce04106f to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304213546ce539b374bbdb7101e43ce04106f 2023-04-21 01:56:41,638 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421436ed227bb8f4e7581388632334690aa to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421436ed227bb8f4e7581388632334690aa 2023-04-21 01:56:41,639 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042147d697e10e084987a9ac3d95027dcf5e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042147d697e10e084987a9ac3d95027dcf5e 2023-04-21 01:56:41,640 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304214cd63ed0a16b42ccaed973328952a37e to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304214cd63ed0a16b42ccaed973328952a37e 2023-04-21 01:56:41,641 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421592719bb82b34d8dbf682e3cdb65bf96 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421592719bb82b34d8dbf682e3cdb65bf96 2023-04-21 01:56:41,642 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421616dfa7db796458093aec5e6901aed66 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421616dfa7db796458093aec5e6901aed66 2023-04-21 01:56:41,643 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304216c09abff88fd4606a08e0882f82b87e6 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304216c09abff88fd4606a08e0882f82b87e6 2023-04-21 01:56:41,644 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304217171015d39e54a9e9c1a61b3dafd30ec to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304217171015d39e54a9e9c1a61b3dafd30ec 2023-04-21 01:56:41,645 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304217dff9456a5a64088aa626b964b4c50fa to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304217dff9456a5a64088aa626b964b4c50fa 2023-04-21 01:56:41,646 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042182f6a92016f14804a315c7d4e8984542 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042182f6a92016f14804a315c7d4e8984542 2023-04-21 01:56:41,647 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304218680123a6de34525b68241d57de906a3 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304218680123a6de34525b68241d57de906a3 2023-04-21 01:56:41,648 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304219a27230ceebb4a3eaadd05cdc0098589 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304219a27230ceebb4a3eaadd05cdc0098589 2023-04-21 01:56:41,649 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421a3de9c23b69c4906b8f53fa856b2ba40 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421a3de9c23b69c4906b8f53fa856b2ba40 2023-04-21 01:56:41,650 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421b43024c842ed45ab91b8c9ded44edcca to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421b43024c842ed45ab91b8c9ded44edcca 2023-04-21 01:56:41,651 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421b6b5a29b282343f7b986e2c3f4c1cd9b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421b6b5a29b282343f7b986e2c3f4c1cd9b 2023-04-21 01:56:41,652 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421bbba56bcfe4e4e8d87ced36267a83c7b to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421bbba56bcfe4e4e8d87ced36267a83c7b 2023-04-21 01:56:41,653 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421bd55f6341e344c059aa19b9e2bacff24 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421bd55f6341e344c059aa19b9e2bacff24 2023-04-21 01:56:41,654 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421cc7097028ba646739f032a229a253257 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421cc7097028ba646739f032a229a253257 2023-04-21 01:56:41,655 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421d19c37ab4b9a4a2384bd6187e146f196 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421d19c37ab4b9a4a2384bd6187e146f196 2023-04-21 01:56:41,656 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421dae349e8acfd4178851e5efce46e9497 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421dae349e8acfd4178851e5efce46e9497 2023-04-21 01:56:41,657 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421dbd321774030424fa320f3bb4f1f7bda to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421dbd321774030424fa320f3bb4f1f7bda 2023-04-21 01:56:41,658 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421f109f03103074365b610e796aad55a79 to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421f109f03103074365b610e796aad55a79 2023-04-21 01:56:41,659 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421f20f74a3717d4d47a6f33009cb87ee3d to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421f20f74a3717d4d47a6f33009cb87ee3d 2023-04-21 01:56:41,660 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421f74972bd03124d76930c013c206196ba to hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230421f74972bd03124d76930c013c206196ba 2023-04-21 01:56:41,661 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-21 01:56:41,662 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:56:41,664 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-21 01:56:41,665 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-21 01:56:41,666 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:56:41,666 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-21 01:56:41,666 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682042201666"}]},"ts":"9223372036854775807"} 2023-04-21 01:56:41,667 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-21 01:56:41,668 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 3db499d69ebc07e344a9c8f0976f4e89, NAME => 'TestAcidGuarantees,,1682042169545.3db499d69ebc07e344a9c8f0976f4e89.', STARTKEY => '', ENDKEY => ''}] 2023-04-21 01:56:41,668 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-21 01:56:41,668 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682042201668"}]},"ts":"9223372036854775807"} 2023-04-21 01:56:41,669 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-21 01:56:41,670 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-21 01:56:41,671 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 63 msec 2023-04-21 01:56:41,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35449] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-21 01:56:41,711 INFO [Listener at localhost/39939] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-21 01:56:41,725 INFO [Listener at localhost/39939] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 230), OpenFileDescriptor=574 (was 563) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=375 (was 422), ProcessCount=169 (was 169), AvailableMemoryMB=9010 (was 8910) - AvailableMemoryMB LEAK? - 2023-04-21 01:56:41,725 INFO [Listener at localhost/39939] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-21 01:56:41,726 INFO [Listener at localhost/39939] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-21 01:56:41,726 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x373f9c99 to 127.0.0.1:58383 2023-04-21 01:56:41,726 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:41,727 DEBUG [Listener at localhost/39939] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-21 01:56:41,727 DEBUG [Listener at localhost/39939] util.JVMClusterUtil(257): Found active master hash=1143794569, stopped=false 2023-04-21 01:56:41,727 INFO [Listener at localhost/39939] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,35449,1682042018145 2023-04-21 01:56:41,728 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-21 01:56:41,728 INFO [Listener at localhost/39939] procedure2.ProcedureExecutor(629): Stopping 2023-04-21 01:56:41,728 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-21 01:56:41,728 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-21 01:56:41,728 DEBUG [Listener at localhost/39939] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2881fe27 to 127.0.0.1:58383 2023-04-21 01:56:41,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-21 01:56:41,728 DEBUG [Listener at localhost/39939] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:41,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-21 01:56:41,729 INFO [Listener at localhost/39939] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,40131,1682042019399' ***** 2023-04-21 01:56:41,729 INFO [Listener at localhost/39939] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-21 01:56:41,729 INFO [RS:0;jenkins-hbase4:40131] regionserver.HeapMemoryManager(220): Stopping 2023-04-21 01:56:41,729 INFO [RS:0;jenkins-hbase4:40131] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-21 01:56:41,729 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-21 01:56:41,729 INFO [RS:0;jenkins-hbase4:40131] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-21 01:56:41,729 INFO [RS:0;jenkins-hbase4:40131] regionserver.HRegionServer(3304): Received CLOSE for cc2eba61dc5e0ef952a659f625b99f73 2023-04-21 01:56:41,730 INFO [RS:0;jenkins-hbase4:40131] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:41,730 DEBUG [RS:0;jenkins-hbase4:40131] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x79c8271d to 127.0.0.1:58383 2023-04-21 01:56:41,731 DEBUG [RS:0;jenkins-hbase4:40131] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:41,731 INFO [RS:0;jenkins-hbase4:40131] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-21 01:56:41,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing cc2eba61dc5e0ef952a659f625b99f73, disabling compactions & flushes 2023-04-21 01:56:41,731 INFO [RS:0;jenkins-hbase4:40131] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-21 01:56:41,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682042021776.cc2eba61dc5e0ef952a659f625b99f73. 2023-04-21 01:56:41,731 INFO [RS:0;jenkins-hbase4:40131] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-21 01:56:41,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682042021776.cc2eba61dc5e0ef952a659f625b99f73. 2023-04-21 01:56:41,731 INFO [RS:0;jenkins-hbase4:40131] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-21 01:56:41,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682042021776.cc2eba61dc5e0ef952a659f625b99f73. after waiting 0 ms 2023-04-21 01:56:41,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682042021776.cc2eba61dc5e0ef952a659f625b99f73. 2023-04-21 01:56:41,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing cc2eba61dc5e0ef952a659f625b99f73 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-21 01:56:41,731 INFO [RS:0;jenkins-hbase4:40131] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-21 01:56:41,731 DEBUG [RS:0;jenkins-hbase4:40131] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, cc2eba61dc5e0ef952a659f625b99f73=hbase:namespace,,1682042021776.cc2eba61dc5e0ef952a659f625b99f73.} 2023-04-21 01:56:41,732 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-21 01:56:41,732 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-21 01:56:41,732 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-21 01:56:41,732 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-21 01:56:41,732 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-21 01:56:41,732 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-21 01:56:41,734 DEBUG [RS:0;jenkins-hbase4:40131] regionserver.HRegionServer(1505): Waiting on 1588230740, cc2eba61dc5e0ef952a659f625b99f73 2023-04-21 01:56:41,739 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/namespace/cc2eba61dc5e0ef952a659f625b99f73/.tmp/info/c0d4e481a55d43dc9bb93a924f2be614 2023-04-21 01:56:41,739 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740/.tmp/info/341b9a5827b644758197fdd3f5acf2b1 2023-04-21 01:56:41,743 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/namespace/cc2eba61dc5e0ef952a659f625b99f73/.tmp/info/c0d4e481a55d43dc9bb93a924f2be614 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/namespace/cc2eba61dc5e0ef952a659f625b99f73/info/c0d4e481a55d43dc9bb93a924f2be614 2023-04-21 01:56:41,752 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 341b9a5827b644758197fdd3f5acf2b1 2023-04-21 01:56:41,753 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/namespace/cc2eba61dc5e0ef952a659f625b99f73/info/c0d4e481a55d43dc9bb93a924f2be614, entries=2, sequenceid=6, filesize=4.8 K 2023-04-21 01:56:41,754 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for cc2eba61dc5e0ef952a659f625b99f73 in 23ms, sequenceid=6, compaction requested=false 2023-04-21 01:56:41,758 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/namespace/cc2eba61dc5e0ef952a659f625b99f73/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-21 01:56:41,765 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682042021776.cc2eba61dc5e0ef952a659f625b99f73. 2023-04-21 01:56:41,765 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for cc2eba61dc5e0ef952a659f625b99f73: 2023-04-21 01:56:41,765 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1682042021776.cc2eba61dc5e0ef952a659f625b99f73. 2023-04-21 01:56:41,768 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-21 01:56:41,786 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-21 01:56:41,786 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-21 01:56:41,934 DEBUG [RS:0;jenkins-hbase4:40131] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-21 01:56:42,134 DEBUG [RS:0;jenkins-hbase4:40131] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-21 01:56:42,167 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740/.tmp/rep_barrier/df0ee1b7cbfe4f3a851bea3527f6e8ca 2023-04-21 01:56:42,170 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for df0ee1b7cbfe4f3a851bea3527f6e8ca 2023-04-21 01:56:42,335 DEBUG [RS:0;jenkins-hbase4:40131] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-21 01:56:42,535 DEBUG [RS:0;jenkins-hbase4:40131] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-21 01:56:42,576 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740/.tmp/table/a10613c0b75a41a4896a38114346eecb 2023-04-21 01:56:42,581 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a10613c0b75a41a4896a38114346eecb 2023-04-21 01:56:42,581 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740/.tmp/info/341b9a5827b644758197fdd3f5acf2b1 as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740/info/341b9a5827b644758197fdd3f5acf2b1 2023-04-21 01:56:42,584 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 341b9a5827b644758197fdd3f5acf2b1 2023-04-21 01:56:42,584 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740/info/341b9a5827b644758197fdd3f5acf2b1, entries=22, sequenceid=93, filesize=7.1 K 2023-04-21 01:56:42,586 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740/.tmp/rep_barrier/df0ee1b7cbfe4f3a851bea3527f6e8ca as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740/rep_barrier/df0ee1b7cbfe4f3a851bea3527f6e8ca 2023-04-21 01:56:42,589 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for df0ee1b7cbfe4f3a851bea3527f6e8ca 2023-04-21 01:56:42,589 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740/rep_barrier/df0ee1b7cbfe4f3a851bea3527f6e8ca, entries=6, sequenceid=93, filesize=5.5 K 2023-04-21 01:56:42,590 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740/.tmp/table/a10613c0b75a41a4896a38114346eecb as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740/table/a10613c0b75a41a4896a38114346eecb 2023-04-21 01:56:42,594 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a10613c0b75a41a4896a38114346eecb 2023-04-21 01:56:42,594 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740/table/a10613c0b75a41a4896a38114346eecb, entries=9, sequenceid=93, filesize=5.5 K 2023-04-21 01:56:42,595 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 863ms, sequenceid=93, compaction requested=false 2023-04-21 01:56:42,599 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-21 01:56:42,599 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-21 01:56:42,601 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-21 01:56:42,601 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-21 01:56:42,601 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-21 01:56:42,735 INFO [RS:0;jenkins-hbase4:40131] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,40131,1682042019399; all regions closed. 2023-04-21 01:56:42,744 DEBUG [RS:0;jenkins-hbase4:40131] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/oldWALs 2023-04-21 01:56:42,744 INFO [RS:0;jenkins-hbase4:40131] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C40131%2C1682042019399.meta:.meta(num 1682042021496) 2023-04-21 01:56:42,748 DEBUG [RS:0;jenkins-hbase4:40131] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/oldWALs 2023-04-21 01:56:42,749 INFO [RS:0;jenkins-hbase4:40131] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C40131%2C1682042019399:(num 1682042020864) 2023-04-21 01:56:42,749 DEBUG [RS:0;jenkins-hbase4:40131] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:42,749 INFO [RS:0;jenkins-hbase4:40131] regionserver.LeaseManager(133): Closed leases 2023-04-21 01:56:42,749 INFO [RS:0;jenkins-hbase4:40131] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-21 01:56:42,749 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-21 01:56:42,751 INFO [RS:0;jenkins-hbase4:40131] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:40131 2023-04-21 01:56:42,758 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-21 01:56:42,758 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,40131,1682042019399 2023-04-21 01:56:42,759 ERROR [Listener at localhost/39939-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7b8d2e50 rejected from java.util.concurrent.ThreadPoolExecutor@7fb2b5f9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2737] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-21 01:56:42,760 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-21 01:56:42,760 ERROR [Listener at localhost/39939-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@192d1b22 rejected from java.util.concurrent.ThreadPoolExecutor@7fb2b5f9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2737] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-21 01:56:42,761 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,40131,1682042019399] 2023-04-21 01:56:42,761 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,40131,1682042019399; numProcessing=1 2023-04-21 01:56:42,763 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,40131,1682042019399 already deleted, retry=false 2023-04-21 01:56:42,763 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,40131,1682042019399 expired; onlineServers=0 2023-04-21 01:56:42,763 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,35449,1682042018145' ***** 2023-04-21 01:56:42,763 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-21 01:56:42,763 DEBUG [M:0;jenkins-hbase4:35449] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@511e1b2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-21 01:56:42,763 INFO [M:0;jenkins-hbase4:35449] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,35449,1682042018145 2023-04-21 01:56:42,763 INFO [M:0;jenkins-hbase4:35449] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,35449,1682042018145; all regions closed. 2023-04-21 01:56:42,763 DEBUG [M:0;jenkins-hbase4:35449] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-21 01:56:42,763 DEBUG [M:0;jenkins-hbase4:35449] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-21 01:56:42,763 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-21 01:56:42,763 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682042020586] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682042020586,5,FailOnTimeoutGroup] 2023-04-21 01:56:42,763 DEBUG [M:0;jenkins-hbase4:35449] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-21 01:56:42,763 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682042020587] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682042020587,5,FailOnTimeoutGroup] 2023-04-21 01:56:42,764 INFO [M:0;jenkins-hbase4:35449] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-21 01:56:42,764 INFO [M:0;jenkins-hbase4:35449] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-21 01:56:42,764 INFO [M:0;jenkins-hbase4:35449] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-21 01:56:42,765 DEBUG [M:0;jenkins-hbase4:35449] master.HMaster(1512): Stopping service threads 2023-04-21 01:56:42,765 INFO [M:0;jenkins-hbase4:35449] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-21 01:56:42,765 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-21 01:56:42,765 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-21 01:56:42,765 ERROR [M:0;jenkins-hbase4:35449] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-21 01:56:42,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-21 01:56:42,766 INFO [M:0;jenkins-hbase4:35449] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-21 01:56:42,766 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-21 01:56:42,766 DEBUG [M:0;jenkins-hbase4:35449] zookeeper.ZKUtil(398): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-21 01:56:42,766 WARN [M:0;jenkins-hbase4:35449] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-21 01:56:42,766 INFO [M:0;jenkins-hbase4:35449] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-21 01:56:42,767 INFO [M:0;jenkins-hbase4:35449] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-21 01:56:42,767 DEBUG [M:0;jenkins-hbase4:35449] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-21 01:56:42,767 INFO [M:0;jenkins-hbase4:35449] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-21 01:56:42,767 DEBUG [M:0;jenkins-hbase4:35449] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-21 01:56:42,767 DEBUG [M:0;jenkins-hbase4:35449] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-21 01:56:42,767 DEBUG [M:0;jenkins-hbase4:35449] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-21 01:56:42,767 INFO [M:0;jenkins-hbase4:35449] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.10 KB heapSize=393.85 KB 2023-04-21 01:56:42,862 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-21 01:56:42,862 INFO [RS:0;jenkins-hbase4:40131] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,40131,1682042019399; zookeeper connection closed. 2023-04-21 01:56:42,862 ERROR [Listener at localhost/39939-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@61373815 rejected from java.util.concurrent.ThreadPoolExecutor@7fb2b5f9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2737] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-21 01:56:42,864 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@149c67a3] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@149c67a3 2023-04-21 01:56:42,864 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): regionserver:40131-0x101a46c3fae0001, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-21 01:56:42,864 ERROR [Listener at localhost/39939-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2b9a2f5e rejected from java.util.concurrent.ThreadPoolExecutor@7fb2b5f9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2737] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-21 01:56:42,865 INFO [Listener at localhost/39939] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-21 01:56:43,185 INFO [M:0;jenkins-hbase4:35449] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.10 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/50d00a68ec924991b6bf901e7411848b 2023-04-21 01:56:43,188 INFO [M:0;jenkins-hbase4:35449] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 50d00a68ec924991b6bf901e7411848b 2023-04-21 01:56:43,188 DEBUG [M:0;jenkins-hbase4:35449] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/50d00a68ec924991b6bf901e7411848b as hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/50d00a68ec924991b6bf901e7411848b 2023-04-21 01:56:43,191 INFO [M:0;jenkins-hbase4:35449] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 50d00a68ec924991b6bf901e7411848b 2023-04-21 01:56:43,191 INFO [M:0;jenkins-hbase4:35449] regionserver.HStore(1080): Added hdfs://localhost:36883/user/jenkins/test-data/e16f4689-221a-e083-371b-3382d9647a58/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/50d00a68ec924991b6bf901e7411848b, entries=65, sequenceid=579, filesize=24.7 K 2023-04-21 01:56:43,191 INFO [M:0;jenkins-hbase4:35449] regionserver.HRegion(2948): Finished flush of dataSize ~345.10 KB/353382, heapSize ~393.84 KB/403288, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 424ms, sequenceid=579, compaction requested=false 2023-04-21 01:56:43,192 INFO [M:0;jenkins-hbase4:35449] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-21 01:56:43,192 DEBUG [M:0;jenkins-hbase4:35449] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-21 01:56:43,195 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-21 01:56:43,195 INFO [M:0;jenkins-hbase4:35449] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-21 01:56:43,196 INFO [M:0;jenkins-hbase4:35449] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:35449 2023-04-21 01:56:43,197 DEBUG [M:0;jenkins-hbase4:35449] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,35449,1682042018145 already deleted, retry=false 2023-04-21 01:56:43,300 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-21 01:56:43,300 INFO [M:0;jenkins-hbase4:35449] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,35449,1682042018145; zookeeper connection closed. 2023-04-21 01:56:43,300 ERROR [Listener at localhost/39939-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@e57deaa rejected from java.util.concurrent.ThreadPoolExecutor@75a0b68f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4569] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-21 01:56:43,300 DEBUG [Listener at localhost/39939-EventThread] zookeeper.ZKWatcher(600): master:35449-0x101a46c3fae0000, quorum=127.0.0.1:58383, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-21 01:56:43,300 ERROR [Listener at localhost/39939-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2e523c7c rejected from java.util.concurrent.ThreadPoolExecutor@75a0b68f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4569] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-21 01:56:43,302 WARN [Listener at localhost/39939] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-21 01:56:43,305 INFO [Listener at localhost/39939] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-21 01:56:43,413 WARN [BP-136531985-172.31.14.131-1682042014715 heartbeating to localhost/127.0.0.1:36883] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-21 01:56:43,413 WARN [BP-136531985-172.31.14.131-1682042014715 heartbeating to localhost/127.0.0.1:36883] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-136531985-172.31.14.131-1682042014715 (Datanode Uuid 42d80742-5c6e-423e-b2cf-6734fc36de52) service to localhost/127.0.0.1:36883 2023-04-21 01:56:43,416 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/cluster_95a6cece-d06d-0ad6-c9d7-e0ee9e760277/dfs/data/data1/current/BP-136531985-172.31.14.131-1682042014715] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-21 01:56:43,417 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9827633c-3a8a-ef79-fe38-7c9861afc6de/cluster_95a6cece-d06d-0ad6-c9d7-e0ee9e760277/dfs/data/data2/current/BP-136531985-172.31.14.131-1682042014715] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-21 01:56:43,455 INFO [Listener at localhost/39939] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-21 01:56:43,629 INFO [Listener at localhost/39939] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-21 01:56:43,818 INFO [Listener at localhost/39939] hbase.HBaseTestingUtility(1293): Minicluster is down